Page 1 of 1
Delayed Rules Blocking Googlebot
Posted: Thu Apr 19, 2012 6:25 pm
by nootkan
Hi all, I am new to using modsecurity v 2.6.3 and the delayed ruleset so please bare with me. I am seeing this in my logs:
2012-04-19 08:31:38 66.249.67.198 /clickmanager/clickmanager.cgi?dl=
http://www.guidesforhire.com HTTP/1.1
www.mywebsite.com Access denied with code 403 (phase 2). Match of "beginsWith http:/%{SERVER_NAME}/" against "MATCHED_VAR" required. [file "/usr/local/apache/conf/modsec/10_asl_rules.conf"] [line "486"] [id "340162"] [rev "261"] [msg "Atomicorp.com UNSUPPORTED DELAYED Rules: Remote File Injection attempt in ARGS (AE)"] [data "http:/"] [severity "CRITICAL"]
The ip address belongs to a googlebot and the link is a click manager I use to keep track of clicks on urls within the website. Is this a normal block for an actual remote file injection or a false postitive and is it actually stopping googlebot from indexing the page/links? If so, how would I allow googlebot to do it's thing?
Re: Delayed Rules Blocking Googlebot
Posted: Fri Apr 20, 2012 2:12 pm
by mikeshinn
That looks like a false positive, its definitely not in the real time rules. So you can either wait for the delayed rules to catch up to the real time rules (approx. 90 days) or just use the real time rules which do not have this issue.
Re: Delayed Rules Blocking Googlebot
Posted: Fri Apr 20, 2012 4:11 pm
by nootkan
Thanks for your reply. I just disabled the rule id as it seems to be creating a lot of false positives. Unfortunately I paid someone to install modsecurity and the delayed rules because I'm a web designer just learning how to manage my own dedicated server and as you probably already know it's no easy chore to learn on my own. I'm not comfortable trying to install the real time rules myself. Time is my friend along with forums and knowledgeable persons like yourself. Again thanks for your help.
Re: Delayed Rules Blocking Googlebot
Posted: Fri Apr 20, 2012 4:15 pm
by mikeshinn
I would not recommend you disable that rule if you use any PHP scripts. 340162 is the generic remote file injection protection rule, disabling that means your system will have no protection against RFI attacks. If you don't use PHP, then you can disable this rule. If you do use PHP, definitely don't disable that rule.
Or just use the real time rules, which do not cause this false positive. You can get a copy of them from the URL below:
https://www.atomicorp.com/products/modsecurity.html
Re: Delayed Rules Blocking Googlebot
Posted: Fri Apr 20, 2012 4:48 pm
by nootkan
Again thanks for your reply, however after reading through the documentation I don't feel comfortable trying to install the real time rules myself and I don't want to mess anything up. It looks like you don't provide an installation service other than one for 299 dollars and seeing how this isn't a commercial project for me yet I cannot afford the cost. My budget has already been reached for setting up my server so I have to try and learn the rest on my own for now. Thanks for your help much appreciated.
I do use php scripts on the server so I will enable the rule again. Are these false positives actually blocked or just a warning? If they are blocked surely there must be away to unblock them. If so, where do I find the documentation on how to unblock them?
Re: Delayed Rules Blocking Googlebot
Posted: Fri Apr 20, 2012 4:57 pm
by mikeshinn
I'm not sure where you found 299 on the website, we definitely don't charge that for installation (if you found that somewhere, please let me know so we can fix that!). Installation prices are $59 for standards based systems (RHEL, Centos, etc. using RPMs) and $99 for custom systems (source builds, etc.)
Our products prices are:
Atomic Secured Linux: $199.95 Annually or $24.95 Monthly
Real Time Web Application Rules subscription (this is included in ASL): $99.99 Annually or $12.95 a month.
Are these false positives actually blocked or just a warning?
That depends on how the person that setup modsecurity configured it. Normally it would block them, but if they didnt setup modsecurity to block anything then its just logging the attacks.
If so, where do I find the documentation on how to unblock them?
Do you mean how would you prevent the rule from being triggered?
Re: Delayed Rules Blocking Googlebot
Posted: Fri Apr 20, 2012 5:37 pm
by nootkan
Sorry I meant the 199.95 package. I saw the reference to the real time rule 99/yr package but it didn't explain whether you installed the rules or I had to so I assumed I had to. Never found the real time rule 59/yr package. Doesn't mean it isn't there however, I could just be skimming over it as I tend to do this when I see jargon or documentation that I don't understand.
That depends on how the person that setup modsecurity configured it. Normally it would block them, but if they didnt setup modsecurity to block anything then its just logging the attacks.
Code: Select all
# ConfigServer ModSecurity whitelist file
Include /usr/local/apache/conf/modsec2.whitelist.conf
SecCacheTransformations Off
SecRuleEngine On
SecRequestBodyAccess On
SecResponseBodyAccess On
SecResponseBodyMimeType (null) text/html text/plain text/xml
SecResponseBodyLimit 2621440
SecArgumentSeparator "&"
SecCookieFormat 0
SecRequestBodyInMemoryLimit 131072
SecResponseBodyLimitAction ProcessPartial
SecUploadDir /usr/local/apache/conf/modsec/data/suspicious
SecTmpDir /tmp
SecDataDir /usr/local/apache/conf/modsec/data/msa
SecServerSignature Apache
SecPcreMatchLimit 100000
SecPcreMatchLimitRecursion 100000
Include /usr/local/apache/conf/modsec/05_asl_exclude.conf
#Include /usr/local/apache/conf/modsec/10_asl_antimalware.conf
Include /usr/local/apache/conf/modsec/10_asl_rules.conf
Include /usr/local/apache/conf/modsec/20_asl_useragents.conf
#Include /usr/local/apache/conf/modsec/30_asl_antispam.conf
#Include /usr/local/apache/conf/modsec/30_asl_antispam_referrer.conf
Include /usr/local/apache/conf/modsec/50_asl_rootkits.conf
Include /usr/local/apache/conf/modsec/60_asl_recons.conf
Include /usr/local/apache/conf/modsec/99_asl_jitp.conf
Do you mean how would you prevent the rule from being triggered?
Sorry again but yes that is what I meant.
Re: Delayed Rules Blocking Googlebot
Posted: Fri Apr 20, 2012 5:57 pm
by mikeshinn
Sorry I meant the 199.95 package. I saw the reference to the real time rule 99/yr package but it didn't explain whether you installed the rules or I had to so I assumed I had to.
ASL will install everything for you, the real time rules only package is something you have to install yourself. Or, if you want us to install it, the service fee is $59 for a standards based system, and $99 for a custom system (this does not include the license fees for either product, which is separate).
Quote:
Do you mean how would you prevent the rule from being triggered?
Sorry again but yes that is what I meant.
If you are using ASL, you can either:
1) Change the rule action from the GUI and you're done.
2) Report is as a false positive to us and we'll fix the rule for you the same day you report it (so it wont trigger incorrectly anymore).
If you are using the real time rules, you can do #2 above.
If you are using the unsupported delayed rules, you'll need to either fix the rule yourself, or you'll need to disable it either globally or by application path. Documentation for disabling rules is available here:
https://www.atomicorp.com/wiki/index.php/Mod_security
Re: Delayed Rules Blocking Googlebot
Posted: Fri Apr 20, 2012 6:34 pm
by nootkan
ASL will install everything for you, the real time rules only package is something you have to install yourself. Or, if you want us to install it, the service fee is $59 for a standards based system, and $99 for a custom system (this does not include the license fees for either product, which is separate).
Thanks, that explanation I understand. I will take you up on the $59 install plus the real time ruleset annual fee of $99 when I go back to work in May but until then I guess I will have to either disable the rule or
If you are using the unsupported delayed rules, you'll need to either fix the rule yourself, or you'll need to disable it either globally or by application path. Documentation for disabling rules is available here:
https://www.atomicorp.com/wiki/index.php/Mod_security
After reading through the wiki can I whitelist the googlebot range: 66.249.64.0/19 in my WHM plugin modsec control panel modsec2.whitelist.conf by placing the ip range inside the file instead of the two options you mention above? Would that work also?
Re: Delayed Rules Blocking Googlebot
Posted: Fri Apr 20, 2012 7:01 pm
by mikeshinn
Depending on how they set that up in WHM it may work, you can also use the examples in the wiki to setup whitelisting, or you can use the 00_asl_whitelist.conf ruleset and just add the range to the /etc/asl/whitelist file. It doesnt look like whomever setup modsecurity for you set that file up, so you'll need to add that ruleset it in and create the /etc/asl/whitelist file.
Re: Delayed Rules Blocking Googlebot
Posted: Fri Apr 20, 2012 7:33 pm
by nootkan
I see this at the top of my modsec config in WHM
# ModSecurity whitelist file
Include /usr/local/apache/conf/modsec2.whitelist.conf
and this also in my modsec plugin control
Doesn't this mean it is set up properly?
you can also use the examples in the wiki to setup whitelisting
This is where I got the idea from.
If you are not using ASL, simply add your IP address to the file:
/etc/asl/whitelist
And restart Apache.
Note: For this rule to work, in ASL you must have the MODSEC_00_WHITELIST ruleset enabled.
If you are not using ASL, then you must have the 00_asl_whitelist.conf ruleset loaded.
# Do not edit this file!
# This file is generated and changes will be overwritten.
This is what I see inside the modsec/00_asl_whitelist.conf when I click the edit button in WHM plugin control so I assume I must add the ip address range in modsec2.whitelist.conf correct?
Udate: I tried to add the ip range into modsec2.whitelist.conf and received this error so you must be right in your assumption on it not being set up right. I have no idea how to set it up right if I cannot do it inside of WHM so I guess I'm hooped.
An error occurred while running: /usr/local/apache/bin/httpd -DSSL -t -f /usr/local/apache/conf/httpd.conf.work.R4H9ZfI5HWddcVLm
Exit signal was: 0
Exit value was: 1
Output was:
---
Syntax error on line 6 of /usr/local/apache/conf/modsec2.whitelist.conf:
Invalid command '66.249.64.0/19', perhaps misspelled or defined by a module not included in the server configuration
Re: Delayed Rules Blocking Googlebot
Posted: Sat Apr 21, 2012 2:30 pm
by mikeshinn
You want to add the IP range to this file:
/etc/asl/whitelist
For the delayed rules you can not use CIDRs, you will have to use a tcpwrappers format:
66.249.64.