mod_security + mod_rewrite with strange behavior

[eduardosilva]

Hello there,

I have a cPanel server with suhosin + mod_security + asl-lite .
All the sites hosted at this server seen to have mod_security working fine. But my blog based in wordpress is having some really weird issues.

The suggested attack from the wiki isn't being blocked by mod_security for my blog. (no 403 error). I can see the block message in log file (saying 403) but I still can access the site. After a lot of testing, I discovered that the default rewrite rules from wordpress at .htaccess are messing with my mod_security setup. If I comment out the rewrite rules, the blocks (403) would work as they should but as soon as they are active my mod_security stops working as it should for this site.

.htaccess

Code: Select all

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

Test attack

Code: Select all

# wget http://eduardosilva.eti.br/foo.php?foo=http://fakeattacker.com
--2011-05-24 23:04:24--  http://eduardosilva.eti.br/foo.php?foo=http://fakeattacker.com
Resolving eduardosilva.eti.br... 96.31.88.180
Connecting to eduardosilva.eti.br|96.31.88.180|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2011-05-24 23:04:24 ERROR 404: Not Found.

root@junglehunt [~/test]# wget http://eduardosilva.eti.br/wp-login.php?foo=http://fakeattacker.com
--2011-05-24 23:04:32--  http://eduardosilva.eti.br/wp-login.php?foo=http://fakeattacker.com
Resolving eduardosilva.eti.br... 96.31.88.180
Connecting to eduardosilva.eti.br|96.31.88.180|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4677 (4.6K) [text/html]
Saving to: `wp-login.php?foo=http:%2F%2Ffakeattacker.com'

100%[===============================================================>] 4,677       --.-K/s   in 0s      

2011-05-24 23:04:33 (558 MB/s) - `wp-login.php?foo=http:%2F%2Ffakeattacker.com' saved [4677/4677]

error_log (I just noticed that for the wp-login.php does not appear in the log as a block.

Code: Select all

[Tue May 24 23:03:03 2011] [error] [client 96.31.88.180] ModSecurity: Access denied with code 403 (phase 2). Match of "beginsWith http:/%{SERVER_NAME}/" against "MATCHED_VAR" required. [file "/usr/local/apache/conf/modsec_rules/10_asl_rules.conf"] [line "482"] [id "340162"] [rev "228"] [msg "Atomicorp.com WAF Rules: Remote File Injection attempt in ARGS (AE)"] [data "http:/"] [severity "CRITICAL"] [hostname "eduardosilva.eti.br"] [uri "/foo.php"] [unique_id "TdxjV2AfWLQAAC45Q6sAAAAE"]
[Tue May 24 23:04:24 2011] [error] [client 96.31.88.180] ModSecurity: Access denied with code 403 (phase 2). Match of "beginsWith http:/%{SERVER_NAME}/" against "MATCHED_VAR" required. [file "/usr/local/apache/conf/modsec_rules/10_asl_rules.conf"] [line "482"] [id "340162"] [rev "228"] [msg "Atomicorp.com WAF Rules: Remote File Injection attempt in ARGS (AE)"] [data "http:/"] [severity "CRITICAL"] [hostname "eduardosilva.eti.br"] [uri "/foo.php"] [unique_id "TdxjqGAfWLQAAC47SCQAAAAG"]

Other sites without those rewrite rules are working fine, those tests result in 403 errors.

Is this normal? Is mod_security still protecting this blog?
Is index.php + mod_rewrite removing the malicious code?
Any other test that I could try to check if mod_security is doing the job properly?

As far as I understand, mod_security should filter the requests before going to mod_rewrite or even going to php files.
I'm feeling a bit lost and confused here, can someone give me any advice?

Thank you

[mikeshinn]

That sounds a module order problem, where something is loaded into apache before mod_security. Or it might be how you have modsecurity itself configured (the error returned for example can have very strange effects on the server and client, which is why 403 is the default)

First, confirm with cpanel the order in which they load modules, particularly if they load or configure anything before modsecurity, as that can cause some strange behavior like this. Also, make sure you are using a config that works correctly, such as ours:

https://www.atomicorp.com/wiki/index.ph ... rity_Rules

As well as a modsec build thats not been tinkered with and may be causing all kinds of weirdness for you (you can download and install supported rpms from the link above, or unsupported source you can build yourself).

[eduardosilva]

Hello mikeshinn, thank you for your input.

I had a very long talk with cPanel support, trying to solve this, and they insist that the module load order doesn't affect this.

...but that request doesn't appear to even be made, as the .htaccess
file is doing its magic here:

RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]


It is highly unlikely that the main index.php file for WordPress
will contain a remote file include issue (index.php?foo=http://site).
In the event that it ever does, the .htaccess rules put forth by the
vendor are redirecting the request right back to index.php anyway,
which appear to be preventing the access.

I'm not sure of how to check the module load order, and after your reply, I made some searches at google and I found this:

One of the consistent bugbears of configuring Apache 1.3 servers is getting the loading order of the modules correct. This is because the order in which the modules are loaded into the server determines the order in which the Apache core calls them to handle requests.

Apache 2 does away with all of this by allowing modules to specify their own ordering preferences. As a result, the ordering problem no longer exists in Apache 2, and neither do the compile-time options to change it.

Reference: http://www.devshed.com/c/a/Apache/Build ... Want-It/4/

I'm using:
apache 2.2 (tested 2.0 too) + mod_security 2.5.13 provided by cPanel - easyapache3

Taking a closer look at httpd.conf (removing comments and empty lines, those are the first lines of the file):

Include "/usr/local/apache/conf/includes/pre_main_global.conf" (empty file)
Include "/usr/local/apache/conf/includes/pre_main_2.conf" (empty file)
LoadModule auth_passthrough_module modules/mod_auth_passthrough.so
LoadModule bwlimited_module modules/mod_bwlimited.so
LoadModule frontpage_module modules/mod_frontpage.so
Include "/usr/local/apache/conf/php.conf" (this file loads suPHP)
Include "/usr/local/apache/conf/modsec2.conf"
Include "/usr/local/apache/conf/includes/errordocument.conf"

Any ideas or suggestions?

Thank you

[scott]

I think you just ruled out this being rule related. That means its something environmental, I suggest you try mikes options.

And I can tell you for a fact that we've bypassed mod_security in the past through the way modules are loaded on 2.x and also if the modsec config isn't setup right (like body support being disabled, or error codes, etc.)

[mikeshinn]

error_log (I just noticed that for the wp-login.php does not appear in the log as a block.

In that example, if I had to guess you are testing the rules on the system itself yes? If so, you need to remove this line from your configuration:

SecRule REMOTE_ADDR "^127.0.0.1$" nolog,allow

That line basically says to ignore anything from the loopback interface. You don't need that with our rules, and I dont recommend it although I believe its standard in the cpanel modsecurity configuration. It creates a hole on the system allowing anything locally to attack the system. Thats one issue you have, but not the root cause of your mod_rewrite problems.

Apache 2 does away with all of this by allowing modules to specify their own ordering preferences. As a result, the ordering problem no longer exists in Apache 2, and neither do the compile-time options to change it.

I think you misunderstand. When modules are loaded, they are also typically configured. Not all the time, but its fairly normal for the complex modules to be loaded and configured at the same time. So I still think its module order, but it may be another module, probably PHP or something being setup before mod_sec is setup. If it makes it easier to understand, module "configuration" order is what you have to look out for. It may also be your modsecurity configuration, so eliminate that variable too. Make sure you have a prestine config using the URL above to our GUI, and a trusted build that has been tampered with.

Any ideas or suggestions?

Sounds like a weird environmental condition, like a module is processing the request and servicing it before mod_security can block it, or it could be your modsec config.

I can't reproduce it on anything, so because this doesnt happen with standard Centos, Redhat, etc. apache and modsecurity builds using our configs, I'd have to say that its something in the differences between your (cpanels?) configuration of modsecurity and the way we do it. We load and configure mod_security before any other modules, and as I said I know the order matters. So I'd start there. Equally, check your modsec config.

I seem to remember seeing something like this years and years ago. It was a big enough deal that way way back in the modsecurity 1.x days I discovered that if you didnt load mod_security early on, and configure before other modules strange things would happen. For some reason I think it might be PHP that does this, but honestly its been many many years, so I just don't recall exactly - I do know we always force modsecurity to load before anything else, and this isn't a problem for us. I think by default in cpanel mod_security loads later, so that seems like a reasonable place to start with your setup. But it could also be your modsec config too, so also check that.

Include "/usr/local/apache/conf/php.conf" (this file loads suPHP)
Include "/usr/local/apache/conf/modsec2.conf"

Try changing that order. I swear I seem to recall that PHP was one of the modules that caused strange things to happen if it was setup earlier than modsec. Something like this:

Include "/usr/local/apache/conf/modsec2.conf"
Include "/usr/local/apache/conf/php.conf" (this file loads suPHP)

My suggestion would be to ask cpanel to double check this. Its the setup of the modules themselves that may be causing this for you. It doesnt happen with the way we do modsecurity with ASLs, and we initialize and configure modsecurity WAY before PHP, PERL, bw modules, proxies, ssl, etc. etc. is configured in Apache, so theres a data point for you.

The one thing I am certain of is that its not the modsecurity rules causing this, no set of rules can cause the condition you have, the rules just can't do that. Its something in apache and its configuration that is doing this. I've seen it happen before, but we resolved the ordering issue so long ago with our apache setups I just dont remember exactly which module(s) can cause this. I want to say PHP, so try moving PHP after modsec, or if you want to save time move modsec before all the other DSO modules in your apache setup.

Also, your modsec config can also do some things like this, so check your config to make sure it matches ours.

Or just use our mod_security build from ASL and be done with it.

[mikeshinn]

As a followup, so I can experiment with your non-atomicorp cpanel modsecurity build, what modules do you have compiled into cpanels apache and which ones are you loading?

[eduardosilva]

Hey mike and scott, thanks for keeping with me in this problem.
I'll be quoting you to make it easier for answers.

As a followup, so I can experiment with your non-atomicorp cpanel modsecurity build, what modules do you have compiled into cpanels apache and which ones are you loading?

loaded Modules:

Code: Select all

root@junglehunt [~]# /usr/local/apache/bin/apachectl -t -D DUMP_MODULES
Loaded Modules:
 core_module (static)
 authn_file_module (static)
 authz_host_module (static)
 authz_groupfile_module (static)
 authz_user_module (static)
 authz_default_module (static)
 auth_basic_module (static)
 auth_digest_module (static)
 include_module (static)
 filter_module (static)
 log_config_module (static)
 log_forensic_module (static)
 logio_module (static)
 env_module (static)
 expires_module (static)
 headers_module (static)
 unique_id_module (static)
 setenvif_module (static)
 proxy_module (static)
 proxy_connect_module (static)
 proxy_ftp_module (static)
 proxy_http_module (static)
 proxy_scgi_module (static)
 proxy_ajp_module (static)
 proxy_balancer_module (static)
 ssl_module (static)
 mpm_prefork_module (static)
 http_module (static)
 mime_module (static)
 dav_module (static)
 status_module (static)
 autoindex_module (static)
 asis_module (static)
 info_module (static)
 suexec_module (static)
 cgi_module (static)
 dav_fs_module (static)
 vhost_alias_module (static)
 negotiation_module (static)
 dir_module (static)
 actions_module (static)
 speling_module (static)
 userdir_module (static)
 alias_module (static)
 rewrite_module (static)
 so_module (static)
 security2_module (shared)
 auth_passthrough_module (shared)
 bwlimited_module (shared)
 frontpage_module (shared)
 suphp_module (shared)
Syntax OK

Just in case, I will send you a private message with a download link for my easyapache profile. Then you just need to load it at cPanel and it will use my configs.

In that example, if I had to guess you are testing the rules on the system itself yes? If so, you need to remove this line from your configuration:

SecRule REMOTE_ADDR "^127.0.0.1$" nolog,allow

That line basically says to ignore anything from the loopback interface. You don't need that with our rules, and I dont recommend it although I believe its standard in the cpanel modsecurity configuration. It creates a hole on the system allowing anything locally to attack the system. Thats one issue you have, but not the root cause of your mod_rewrite problems.

I have the same issue using any remote client.
For example, you may try to access http://eduardosilva.eti.br/wp-login.php ... tacker.com and you will be able to reach the login form.

The one thing I am certain of is that its not the modsecuity rules causing this, no set of rules can cause the condition you have, the rules just can't do that. Its something in apache and its configuration that is doing this. I've seen it happen before, but we fixed it so long ago with our apache setups I just dont remember exactly which module(s) cause this. I want to say PHP, so try moving PHP after modsec, or if you want to save time move modsec before all the other DSO modules in your apache setup.

I moved the include line to the top of httpd.conf, after restarting apache, I made a few tests and got the same results, I agree with you, that the problem must be related to the module order, it really looks like apache is processing the rewrite before modsec.

Or just use our mod_security build from ASL and be done with it.

I have a vz container running a trial version of ASL, are these the steps that I should take?
- run easy apache at cpanel and remove mod_security support from it, build apache again.
- grab mod_security2.so from ASL box and copy to cpanel/apache modules
- edit the httpd.conf (template), and insert the include for mod_security
Am I missing anything? are those steps correct?
I will try this tomorrow in the morning, I need to sleep soon. I have a good feeling about this option.

Thank you very much for all your effort.

[eduardosilva]

Hello again,

I made more tests during the weekend using the mod_security from asl and also compiled mod_security 2.6 from source.

I've got the same results using those versions.

Thank you.

[mikeshinn]

and also compiled mod_security 2.6 from source.

Right, so you can rule out modsecurity as the source of your problems. As I said previously your problem is not with modsecurity, its with your apache configuration (including your modsecurity configuration). If I recall, you said the rules work with one domain, and not another and only if you setup mod_rewrite. If thats true, then you can rule out the rules too. Its a condition brought on by your apache configuration (or setup).

If you havent had any luck with the cpanel guys, then you have two options:

1) Remove your modsec config, all the configs and install ours. That way you can rule out a modsec config issue.

2) If after restarting apache, it still happens, put apache into a debug mode and see what its doing, and why its doing it. You've got a lot of variables there that you need to isolate. I can tell you the rules are not the source, and modsecurity itself isn't either (although the config could be). It sounds like a configuration issue with apache because I can't reproduce this with a standard Redhat Apache setup.

3) Use a supported modsecurity setup (ASL).

I made more tests during the weekend using the mod_security from asl

Just out of curiosity where did you get this? And how did you install it? cpanel doesnt work with rpms, and we only publish our versions of mod_security as rpms (and you need a license for ASL to download the ASL version, which I don't believe you have?). I know you can't install mod_security via an rpm with cpanel, it wont even work.

So how did you install this and where did you download it?

Are you sure you're not talking about the atomic channel version? If so, thats completely different from what we use in ASL. And even then, we only publish rpms and cpanel wont let you install apache rpms. So what did you install and how did you do it?

OK, so I've setup the same configuration you have on a normal Centos box with cpanel, and I can't reproduce this. So best guess is its something different in your cpanel setup from the default. The rules work just fine with mod_rewrite enabled:

[root@server1 html]# cat .htaccess
RewriteEngine On

RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]

wget --no-check-certificate https://localhost/foo.php?foo=http://www.example.com
HTTP request sent, awaiting response... 403 Forbidden
2011-05-28 17:16:05 ERROR 403: Forbidden.


(This system only allows https)

[modsecurity] [client 127.0.0.1] [domain localhost] [403] [/20110528/20110528-1716/20110528-171605-j@-6On8AAAEAACgX6A8AAAAG] [file "/etc/httpd/modsecurity.d/10_asl_rules.conf"] [line "482"] [id "340162"] [rev "228"] [msg "Atomicorp.com WAF Rules: Remote File Injection attempt in ARGS (AE)"] [data "http:/"] [severity "CRITICAL"] Access denied with code 403 (phase 2). Match of "beginsWith http:/%{SERVER_NAME}/" against "MATCHED_VAR" required.

So the rules work. I really do think this a configuration issue with apache that is either unique to your system, or perhaps in the way cpanel sets up mod_security or other modules. I suggest you remove the cpanel modsec config, replace it with ours and see if that resolves it for you. If that doesn't, put apache in debug mode and see whats going on.

Or install ASL and don't worry about it anymore.