Apache child processes not dying/timing out

[jas8522]

This is most likely not ART distro specific, but since everything I've got on the box is from there (and I can't upgrade components without the good likelihood of incompatibility), I figured I'd try here and see if anyone has a solution.

A few days ago, apache started to continuously spawn processes without killing older child processes. When I restart or stop apache, it shows that a number of child processes did not die from a standard SIGTERM and had to be forced to die. I tried updating apache to 2.0.46-61 from 2.0.46-56 in an attempt to fix it, but I got the same problem (I have since rolled it back to -56). Following the update, I installed PHP5 thinking maybe it was a PHP incompatibility with the new version of Apache. Same problem.

What's really aggravating me is that there is nothing in the logs about it! Nothing is notifying me when the Apache processes hang and no longer respond.

Does anyone have any idea what could be causing this? Perhaps a setting somewhere, or config variable?

Thanks for any help you can provide,

Jordan

[scott]

using any external apache DSOs? cold fusion, mono, mod_security, mod_rewrite, etc?

[jas8522]

Hi Scott,

Only one currently in use is mod_security. I tried comparing the number of blocked items added to the audit log with the amount of processes that become unresponsive within a certain timeframe (say 1 hour), and it didn't match up, but it's possible my ruleset is preventing logging of the rule in question.

Using apache's status page (which I only recently discovered), I have been able to narrow the problem down to what appears to be a problem with SpamKarma for WordPress. One of my customers (luckily this one is a friend of mine) recently updated WordPress to 2.0 which includes SpamKarma. When the visitor to the site attempts to POST a comment or trackback, it just sits there and never finishes. Amazingly this does not provoke any kind of Timeout, neither with PHP nor with Apache's timeouts!

I'm still in the midst of verifying this, and it could be one of the mod_security rules blocking the POST action (although my own comment posts work fine, and don't freeze the process). I've also tried two different rulesets - the one I'm using now from The Planet's support forums, and one that is much lighter which only blocks the basics, and both were causing the same problem. Becuase of this I'm not so sure mod_security is at fault; although I'm certainly not going to rule it out.

Will update when I have further information!

Jordan

[jas8522]

Well that narrows that down, just very surprised that the process never times out!

Jordan

[scott]

If you're an ASL customer you can send a message to support@atomicorp.com. Im not the mod_security guy in the group, so I cant be much help here. I just did the the kernel and the packaging on the mod_security/mod_evasive rpms.

[jdaustin]

My load has been going through the roof over the last week since I installed the mod_security stuff from atomiccorp.com. I just removed mod_security.. load dropped to normal.
I did a yum update.. which picked up a new version of mod_security.. installed it again.. load went from .05 to 3.2 in under a minute. I uninstalled it and load dropped to normal.

[breun]

I never got mod_security to work on Plesk servers without all RAM and swap space being eaten away. Do you see this too?

http://gotroot.com/tiki-view_forum_thre ... rentId=658

[scott]

We figure theres some kind of wierd regex bug somewhere. We run every rule on the ART/Gotroot.com site, plus a whole slew of unpublished ones. I have had that same problem pop up on other systems, and all we've been able to narrow it down to is that on some systems, when using really specific applications, the blacklists cause the box to consume all the memory.

The fix we came up with is in ASL 2.0 (coming real soon), which pushes blacklisting into an RBL.

[BDMM]

I never got mod_security to work on Plesk servers without all RAM and swap space being eaten away. Do you see this too?

I found that if you disable blaclist2.conf and useragents.conf and proxy.conf it brings it down to an acceptable level.

The problem with the gotroot conf files is that there are too many rules which hogs cpu.

[scott]

Well no, its not that its too many rules. We run way more on this server than we have up on gotroot.com. Its a bug in the webserver, or some library that is causing a problem. We run around 25,000 rules here, and I think we've only got about 20,000 of those on gotroot.com on a rh9 box with 1.5G of ram.

[Highland]

This explains the strange state I found my slower box in yesterday. RAM was maxed and nearly 2GB swap file was full. Processor was at an all time high of 24! Had to reboot the box using the provider control panel because shotdown command left the box stuck. This morning I find 500MB of RAM usage. I just removed mod_security for the time being

[scott]

Yep, thats the bug. If you're an ASL user, support@atomicorp.com can help you out with it, I'm not the mod_security guy

[sOliver]

Hi,

sorry for bumping an old thread, but I'm having the same problem with a popular Blog software.
I have like 400 child processes and they just dont die, then the disk IO goes up because the swap file is 2GB large (max) .. could this be an error due to incorrect ASL rules? It's normal for me to have many sleeping processes, but not the large swap file.

Also, please check your POST rules again. When I activate your rules and post a comment it's really, really slow. When I deactivate it it's a lot faster.

Best
Oliver

[scott]

As I recall there was a bug earlier in the httpd 2.2.3 chain that caused that. Make sure you've upgraded to the latest from centos/rhel which is currently:
httpd-2.2.3-43.el5.centos.3

[mikeshinn]

Also, please check your POST rules again. When I activate your rules and post a comment it's really, really slow. When I deactivate it it's a lot faster.

Can you explain what you mean by "POST rules"? If you mean our modsecurity rules, which version of the rules are you running? And whats your systems architecture (RAM, CPU, etc.)