Delayed Rules Blocking Googlebot

Community support forums for the free/delayed modsecurity rules feed. There is no such thing as a bad question here as long as it pertains to using the delayed modsecurity rules feed. Newbies feel free to get help getting started or asking questions that may be obvious.
nootkan
Forum User
Forum User
Posts: 11
Joined: Tue Nov 15, 2011 7:16 pm
Location: Vancouver BC Canada

Delayed Rules Blocking Googlebot

Unread post by nootkan »

Hi all, I am new to using modsecurity v 2.6.3 and the delayed ruleset so please bare with me. I am seeing this in my logs:
2012-04-19 08:31:38 66.249.67.198 /clickmanager/clickmanager.cgi?dl=http://www.guidesforhire.com HTTP/1.1 www.mywebsite.com Access denied with code 403 (phase 2). Match of "beginsWith http:/%{SERVER_NAME}/" against "MATCHED_VAR" required. [file "/usr/local/apache/conf/modsec/10_asl_rules.conf"] [line "486"] [id "340162"] [rev "261"] [msg "Atomicorp.com UNSUPPORTED DELAYED Rules: Remote File Injection attempt in ARGS (AE)"] [data "http:/"] [severity "CRITICAL"]


The ip address belongs to a googlebot and the link is a click manager I use to keep track of clicks on urls within the website. Is this a normal block for an actual remote file injection or a false postitive and is it actually stopping googlebot from indexing the page/links? If so, how would I allow googlebot to do it's thing?
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: Delayed Rules Blocking Googlebot

Unread post by mikeshinn »

That looks like a false positive, its definitely not in the real time rules. So you can either wait for the delayed rules to catch up to the real time rules (approx. 90 days) or just use the real time rules which do not have this issue.
nootkan
Forum User
Forum User
Posts: 11
Joined: Tue Nov 15, 2011 7:16 pm
Location: Vancouver BC Canada

Re: Delayed Rules Blocking Googlebot

Unread post by nootkan »

Thanks for your reply. I just disabled the rule id as it seems to be creating a lot of false positives. Unfortunately I paid someone to install modsecurity and the delayed rules because I'm a web designer just learning how to manage my own dedicated server and as you probably already know it's no easy chore to learn on my own. I'm not comfortable trying to install the real time rules myself. Time is my friend along with forums and knowledgeable persons like yourself. Again thanks for your help.
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: Delayed Rules Blocking Googlebot

Unread post by mikeshinn »

I would not recommend you disable that rule if you use any PHP scripts. 340162 is the generic remote file injection protection rule, disabling that means your system will have no protection against RFI attacks. If you don't use PHP, then you can disable this rule. If you do use PHP, definitely don't disable that rule.

Or just use the real time rules, which do not cause this false positive. You can get a copy of them from the URL below:

https://www.atomicorp.com/products/modsecurity.html
nootkan
Forum User
Forum User
Posts: 11
Joined: Tue Nov 15, 2011 7:16 pm
Location: Vancouver BC Canada

Re: Delayed Rules Blocking Googlebot

Unread post by nootkan »

Again thanks for your reply, however after reading through the documentation I don't feel comfortable trying to install the real time rules myself and I don't want to mess anything up. It looks like you don't provide an installation service other than one for 299 dollars and seeing how this isn't a commercial project for me yet I cannot afford the cost. My budget has already been reached for setting up my server so I have to try and learn the rest on my own for now. Thanks for your help much appreciated.

I do use php scripts on the server so I will enable the rule again. Are these false positives actually blocked or just a warning? If they are blocked surely there must be away to unblock them. If so, where do I find the documentation on how to unblock them?
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: Delayed Rules Blocking Googlebot

Unread post by mikeshinn »

I'm not sure where you found 299 on the website, we definitely don't charge that for installation (if you found that somewhere, please let me know so we can fix that!). Installation prices are $59 for standards based systems (RHEL, Centos, etc. using RPMs) and $99 for custom systems (source builds, etc.)

Our products prices are:

Atomic Secured Linux: $199.95 Annually or $24.95 Monthly

Real Time Web Application Rules subscription (this is included in ASL): $99.99 Annually or $12.95 a month.
Are these false positives actually blocked or just a warning?
That depends on how the person that setup modsecurity configured it. Normally it would block them, but if they didnt setup modsecurity to block anything then its just logging the attacks.
If so, where do I find the documentation on how to unblock them?
Do you mean how would you prevent the rule from being triggered?
nootkan
Forum User
Forum User
Posts: 11
Joined: Tue Nov 15, 2011 7:16 pm
Location: Vancouver BC Canada

Re: Delayed Rules Blocking Googlebot

Unread post by nootkan »

Sorry I meant the 199.95 package. I saw the reference to the real time rule 99/yr package but it didn't explain whether you installed the rules or I had to so I assumed I had to. Never found the real time rule 59/yr package. Doesn't mean it isn't there however, I could just be skimming over it as I tend to do this when I see jargon or documentation that I don't understand.
That depends on how the person that setup modsecurity configured it. Normally it would block them, but if they didnt setup modsecurity to block anything then its just logging the attacks.

Code: Select all

# ConfigServer ModSecurity whitelist file
Include /usr/local/apache/conf/modsec2.whitelist.conf

SecCacheTransformations Off
SecRuleEngine On
SecRequestBodyAccess On
SecResponseBodyAccess On
SecResponseBodyMimeType (null) text/html text/plain text/xml
SecResponseBodyLimit 2621440
SecArgumentSeparator "&" 
SecCookieFormat 0
SecRequestBodyInMemoryLimit 131072
SecResponseBodyLimitAction ProcessPartial
SecUploadDir /usr/local/apache/conf/modsec/data/suspicious
SecTmpDir /tmp
SecDataDir /usr/local/apache/conf/modsec/data/msa
SecServerSignature Apache
SecPcreMatchLimit 100000
SecPcreMatchLimitRecursion 100000

Include /usr/local/apache/conf/modsec/05_asl_exclude.conf
#Include /usr/local/apache/conf/modsec/10_asl_antimalware.conf
Include /usr/local/apache/conf/modsec/10_asl_rules.conf
Include /usr/local/apache/conf/modsec/20_asl_useragents.conf
#Include /usr/local/apache/conf/modsec/30_asl_antispam.conf
#Include /usr/local/apache/conf/modsec/30_asl_antispam_referrer.conf
Include /usr/local/apache/conf/modsec/50_asl_rootkits.conf
Include /usr/local/apache/conf/modsec/60_asl_recons.conf
Include /usr/local/apache/conf/modsec/99_asl_jitp.conf
Do you mean how would you prevent the rule from being triggered?
Sorry again but yes that is what I meant.
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: Delayed Rules Blocking Googlebot

Unread post by mikeshinn »

Sorry I meant the 199.95 package. I saw the reference to the real time rule 99/yr package but it didn't explain whether you installed the rules or I had to so I assumed I had to.
ASL will install everything for you, the real time rules only package is something you have to install yourself. Or, if you want us to install it, the service fee is $59 for a standards based system, and $99 for a custom system (this does not include the license fees for either product, which is separate).
Quote:
Do you mean how would you prevent the rule from being triggered?

Sorry again but yes that is what I meant.
If you are using ASL, you can either:

1) Change the rule action from the GUI and you're done.
2) Report is as a false positive to us and we'll fix the rule for you the same day you report it (so it wont trigger incorrectly anymore).

If you are using the real time rules, you can do #2 above.

If you are using the unsupported delayed rules, you'll need to either fix the rule yourself, or you'll need to disable it either globally or by application path. Documentation for disabling rules is available here:

https://www.atomicorp.com/wiki/index.php/Mod_security
nootkan
Forum User
Forum User
Posts: 11
Joined: Tue Nov 15, 2011 7:16 pm
Location: Vancouver BC Canada

Re: Delayed Rules Blocking Googlebot

Unread post by nootkan »

ASL will install everything for you, the real time rules only package is something you have to install yourself. Or, if you want us to install it, the service fee is $59 for a standards based system, and $99 for a custom system (this does not include the license fees for either product, which is separate).
Thanks, that explanation I understand. I will take you up on the $59 install plus the real time ruleset annual fee of $99 when I go back to work in May but until then I guess I will have to either disable the rule or
If you are using the unsupported delayed rules, you'll need to either fix the rule yourself, or you'll need to disable it either globally or by application path. Documentation for disabling rules is available here:

https://www.atomicorp.com/wiki/index.php/Mod_security
After reading through the wiki can I whitelist the googlebot range: 66.249.64.0/19 in my WHM plugin modsec control panel modsec2.whitelist.conf by placing the ip range inside the file instead of the two options you mention above? Would that work also?
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: Delayed Rules Blocking Googlebot

Unread post by mikeshinn »

Depending on how they set that up in WHM it may work, you can also use the examples in the wiki to setup whitelisting, or you can use the 00_asl_whitelist.conf ruleset and just add the range to the /etc/asl/whitelist file. It doesnt look like whomever setup modsecurity for you set that file up, so you'll need to add that ruleset it in and create the /etc/asl/whitelist file.
nootkan
Forum User
Forum User
Posts: 11
Joined: Tue Nov 15, 2011 7:16 pm
Location: Vancouver BC Canada

Re: Delayed Rules Blocking Googlebot

Unread post by nootkan »

I see this at the top of my modsec config in WHM
# ModSecurity whitelist file
Include /usr/local/apache/conf/modsec2.whitelist.conf
and this also in my modsec plugin control

Image

Doesn't this mean it is set up properly?
you can also use the examples in the wiki to setup whitelisting
This is where I got the idea from.
If you are not using ASL, simply add your IP address to the file:

/etc/asl/whitelist

And restart Apache.

Note: For this rule to work, in ASL you must have the MODSEC_00_WHITELIST ruleset enabled.

If you are not using ASL, then you must have the 00_asl_whitelist.conf ruleset loaded.
# Do not edit this file!
# This file is generated and changes will be overwritten.
This is what I see inside the modsec/00_asl_whitelist.conf when I click the edit button in WHM plugin control so I assume I must add the ip address range in modsec2.whitelist.conf correct?

Udate: I tried to add the ip range into modsec2.whitelist.conf and received this error so you must be right in your assumption on it not being set up right. I have no idea how to set it up right if I cannot do it inside of WHM so I guess I'm hooped.
An error occurred while running: /usr/local/apache/bin/httpd -DSSL -t -f /usr/local/apache/conf/httpd.conf.work.R4H9ZfI5HWddcVLm
Exit signal was: 0
Exit value was: 1
Output was:
---
Syntax error on line 6 of /usr/local/apache/conf/modsec2.whitelist.conf:
Invalid command '66.249.64.0/19', perhaps misspelled or defined by a module not included in the server configuration
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: Delayed Rules Blocking Googlebot

Unread post by mikeshinn »

You want to add the IP range to this file:

/etc/asl/whitelist

For the delayed rules you can not use CIDRs, you will have to use a tcpwrappers format:

66.249.64.
Post Reply