Multiple CL/Centos6 + cPanel Servers crashing

Community support forums for the free/delayed modsecurity rules feed. There is no such thing as a bad question here as long as it pertains to using the delayed modsecurity rules feed. Newbies feel free to get help getting started or asking questions that may be obvious.
mrmason
New Forum User
New Forum User
Posts: 1
Joined: Wed Jun 20, 2012 12:38 pm
Location: United Kingdom

Multiple CL/Centos6 + cPanel Servers crashing

Unread post by mrmason »

Hello,

We've encountered a problem which seems to stem from using your latest delayed rules in conjunction with CentOs & Cloud Linux 6 + cPanel.

We've had the same problem on severs with Centos6+CloudLinux6, but not on CL/Centos5.

Here is our setup:

- Physical servers with 32GB of memory, no virtualisation.
- Cloud Linux6 64bit or Centos 6 64bit.
- Latest version of cPanel with apache build to latest version via EasyApache
- Using Mod_Sec from easy apache

And here is the problem:

When mod_sec is enabled, after a random period of time (usually less than a day), the server totally crashes due to running out of memory.
Before the server fully dies, a snapshot of top shows one httpd process using up a massive amount of memory.

For example:

One server, this was taken about 30 seconds before it crashed:

Code: Select all

267017 nobody    20   0 12.5g 6.7g 2884 R 96.3 21.5   0:10.21 httpd   
And another totally different server: (note that magnet is nothing more than an html website, no php, no database, just html)

Code: Select all

387914 magnet    20   0 47.1g  25g  296 S  0.0 81.3   0:53.60 httpd
If we disable mod_sec totally then the servers don't crash. We also don't seem to have the problem when using the standard cPanel issued rules.

We were hoping to roll out full ASL on some servers soon, but we've been unable to stop these servers from crashing so are worried about making the next move.

Do you have any suggestions on where to start looking?
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: Multiple CL/Centos6 + cPanel Servers crashing

Unread post by mikeshinn »

Sorry to hear about that. If I understand you correctly, as you said you were using the delayed rules, I'm going to assume that you are not using ASL on any of these systems? Just the rules?

If thats the case, that sounds like a bad configuration, overlapping/conflicting third party rules, a leak in apache (or one of its libraries, like APR, libpcre or libxml) or in cpanels modsecurity.

So heres what I would start with:

1) Make sure you are using our modsecurity builds. Cpanel builds theres in some suboptimal ways. Also, the last time I looked cpanel was only up to 2.6.5, which has bugs and a rather large vulnerability. You want to use at least 2.6.6 for those reasons alone. (And the vulnerability will let the bad guys bypass modsecurity, so upgrade!)

2) Use the real time rules. The delayed rules are not supported. You can try the real time rules out for free from this URL:

https://www.atomicorp.com/amember/signu ... aysys=free

3) Make sure your modsecurity configuration exactly matches the configuration described on the wiki. I've definitely seen cases like your where a bad configuration has caused a rule loop (where a configuration is loading multiple rules over and over again). The configuration is the easiest thing to fix, so make sure it matches exactly.

https://www.atomicorp.com/wiki/index.ph ... rity_Rules

4) If you are using any third party rules (including cpanels rules), or addons (like ESX), disable them.

5) Check with cloudlinux as well. Cloudlinux controls all of the things that could be also contributing to this (apache and the kernel). As they make changes to apache and the kernel that effects memory usage and garbage collection this could be a bug. I havent heard of any, so I dont think thats the case here - but you should ask just in case.

modsecurity may just be a symptom of a larger problem with a leak. modsecurity will use up memory, and if you have a broken apache, kernel, or library it will manifest this leak faster. Disabling modsecurity may not actually be removing the problem, it might just be slower with less memory intensive modules.

6) Do you have any core files for apache? If the apache processes are dying off, it sounds like possible faults and you might have core files you can look at to find the real root cause. See the link before for how to set this up (for cpanel, you'll need to contact them about getting the debuginfo for their source built apache)

https://www.atomicorp.com/wiki/index.php/Apache

A backtrace would be really handy to see whats causing apache to die.

7) Have you tried strace on your apache processes to see whats actually causing this with apache using up so much memory?

I've seen a broken PHP application do this. It was calling something that just didnt agree with Apache, and caused a leak to explode. Turning of modsecurity seemed to make it go away, but it turned out it just took longer with it off and mosec wasnt the cause.

8) bad mod_rewrite rules can also cause weird memory problems, I've definitely seen that with the cpanel modsecurity module. I'm not sure what they do, but it causes some odd interactions. You can rule this out by disabling mod_rewrite, and that doesnt solve it you can rule out mod_rewrite/mod_security interactions.
We were hoping to roll out full ASL on some servers soon, but we've been unable to stop these servers from crashing so are worried about making the next move.
Installing ASL is actually the simplest way to fix this problem. Just install ASL, or an ASL trial. ASL will "sanitize" your modsecurity setup with a clean correct working config, and a solid correctly built and optimized modsecurity module plus all the supporting libraries. Its a quick cheat, and you can always uninstall ASL if you dont need it. I've never seen this happen with ASL, but if were the first time we'll get it fixed for you, right away.

If this is still happening you'll know its modsecurity, plus we'll be able to help you with this. We'll even log into your server, for free, to find out whats going and get it fixed. What could be easier or cheaper than that?

With one step you can rule in or out modsecurity as the true cause, plus you'll get rapid support and a supported solution if the problem continues. Like I said, we'll even log into your system, for free, if necessary to find out and fix whatever is causing this.

This is the quickest and easiest way to solve your problem.

You can get a free trial copy here:

https://www.atomicorp.com/amember/signu ... aysys=free
mrsant
Forum User
Forum User
Posts: 17
Joined: Thu Jun 21, 2012 5:07 am
Location: UK

Re: Multiple CL/Centos6 + cPanel Servers crashing

Unread post by mrsant »

Thanks for your comprehensive reply.

I can confirm that our installation matches the wiki instructions precisely. The /var/asl directory structure and permissions/ownerships are all concordant with the standard cpanel/apache nobody account etc.

The modsec2.user.conf and modsec2.conf files match with those recommended. We have disabled the antispam rules, but other than that it's all stock.

Also, it appears apache, and modsec are using the same lib

# ldd /usr/local/apache/modules/mod_security2.so | grep pcre
libpcre.so.0 => /opt/pcre/lib/libpcre.so.0 (0x00007f3c13f03000)
# ldd /usr/local/apache/bin/httpd | grep pcre
libpcre.so.0 => /opt/pcre/lib/libpcre.so.0 (0x00007fa974670000)

Interestingly, in the past 3 days, these are the rules and the number of incidences we have logged in error_log

grep "PCRE limits exceeded" /usr/local/apache/logs/error_log | grep -Eo "\[id[^]]*\"\]" | sort | uniq -c
69 [id "-"] <---- What is this?!
6 [id "340018"]
2 [id "340145"]
62406 [id "340157"] <---- whoa - I whitelisted this one - upon inspection it's a falsie..
3 [id "380025"]
2 [id "390149"]
2 [id "390572"]
69 [id "390801"]
166 [id "390804"]

In your oppinion, could the high number of incidences with rule 340157 have caused, or accelerated another latent fault that (as you point out) may not be directly related to modsec?

I appreciate you would prefer we simply buy ASL (you make a good salesman!), but with the number of servers we are looking to roll out, we would clearly be prudent to find a workable solution using your realtime rules and a suitable version of modsec. So, in that sense, are you aware of any successful implementations of cpanel/centos6/modsec/aslrules where modsec has been manually built and included as an apache module?
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: Multiple CL/Centos6 + cPanel Servers crashing

Unread post by mikeshinn »

grep "PCRE limits exceeded" /usr/local/apache/logs/error_log | grep -Eo "\[id[^]]*\"\]" | sort | uniq -c
So that means you aren't using a properly built modsecurity, that may be whats causing it - regardless though its breaking your install in lots of other ways (like skipping entire attacks and just letter then through happily). Make sure you are using our modsecurity builds, ours cant not generate that error.

Also, have you run an strace and setup your system for core dumps? That kind of information is what you (and we) really need to diagnose a memory problem with Apache.

Also, we highly recommend you contant Cloudlinux and ask them to take a look, as I said before because you are using a modified version of Apache from them there could be a bug in it (or in another library) thats creating the conditions for a memory leak.
Post Reply