Apache bus error with mod_security, httpd-debuginfo missing?

General Discussion of atomic repo and development projects.

Ask for help here with anything else not covered by other forums.
breun
Long Time Forum Regular
Long Time Forum Regular
Posts: 2813
Joined: Sat Aug 20, 2005 9:30 am
Location: The Netherlands

Apache bus error with mod_security, httpd-debuginfo missing?

Unread post by breun »

We have one server running CentOS 5.6, Plesk 8.6.0 and ASL 3.0.3 which reports bus errors (logged as notices) in /var/log/httpd/error_log very frequently, like these:

Code: Select all

[Sat Jul 30 12:36:00 2011] [notice] child pid 24282 exit signal Bus error (7)
[Sat Jul 30 12:36:02 2011] [notice] child pid 24521 exit signal Bus error (7)
[Sat Jul 30 12:36:03 2011] [notice] child pid 24537 exit signal Bus error (7)
[Sat Jul 30 12:36:55 2011] [notice] child pid 24538 exit signal Bus error (7)
[Sat Jul 30 13:11:05 2011] [notice] child pid 30486 exit signal Bus error (7)
[Sat Jul 30 13:11:05 2011] [notice] child pid 30599 exit signal Bus error (7)
[Sat Jul 30 13:11:07 2011] [notice] child pid 30619 exit signal Bus error (7)
When we disable mod_security these bus errors disappear.

I tried to follow the instructions on http://www.atomicorp.com/wiki/index.php/Apache to setup Apache for core dumps, but the CentOS repositories don't seem to have the httpd-debuginfo package mentioned there. Any ideas on how to proceed?
Lemonbit Internet Dedicated Server Management
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: Apache bus error with mod_security, httpd-debuginfo miss

Unread post by mikeshinn »

The art builds all have the debug-info, I seem to recall the centos project was missing theirs. Try our http build, its entirely possible this error is being caused by the known APR bug in older apache builds (such as centos 5 has), or it could be something else entriely. At the very least if a newer apache doesnt resolve that you could rule out APR as the cause, and running a new apache (even temporarily) will let you get a decent trace on apache when this error occurs.
breun
Long Time Forum Regular
Long Time Forum Regular
Posts: 2813
Joined: Sat Aug 20, 2005 9:30 am
Location: The Netherlands

Re: Apache bus error with mod_security, httpd-debuginfo miss

Unread post by breun »

Apache 2.2.17 from atomic-testing has logged no bus error yet, so that's looking good. We'd rather not run testing packages in production though, if at all possible. What's the status of these packages exactly? Really still in testing? Maintained?
Lemonbit Internet Dedicated Server Management
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: Apache bus error with mod_security, httpd-debuginfo miss

Unread post by mikeshinn »

Apache 2.2.17 from atomic-testing has logged no bus error yet, so that's looking good.
I'm not surprised, what lead me to write article on segfaults was a debugging session I did with a state government customer where we discovered it was a bug in APR, and not in modsecurity. We confirmed this when we ran 2.2.17 as a test and sure enough, problem solved. Something is wonky with APR in the stable RHEL 5 and Centos 5 universe, its one of those odd segfaults that rarely happens (for example, it hasnt ever happened on our RHEL 5 and Centos 5 boxes) but when it does, it just does it in droves.
We'd rather not run testing packages in production though, if at all possible. What's the status of these packages exactly? Really still in testing? Maintained?
Great question, not in testing, actively maintained and stable for a long time. We just havent moved them to stable yet. :-)
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8355
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Re: Apache bus error with mod_security, httpd-debuginfo miss

Unread post by scott »

I was looking at updating it to 2.2.19 for a final release to stable.
ikkk
Forum User
Forum User
Posts: 47
Joined: Wed Jan 05, 2011 3:09 pm

Re: Apache bus error with mod_security, httpd-debuginfo miss

Unread post by ikkk »

We have seen the exact errors that breun has posted above on the httpd 2.2.17 from the testing repo , we initially thought that it was the fact that this was a test release that this was happening, but got the same faults in the standard centos http as well.

We found it was happening when trying to do authentication a lot.

We narrowed it down to a rule in the antimalware rules, i cant recall which one exactly now. But on the particular server (only running one domain) that this was on, this issue has not came back after disabling the rule.
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: Apache bus error with mod_security, httpd-debuginfo miss

Unread post by mikeshinn »

We narrowed it down to a rule in the antimalware rules, i cant recall which one exactly now. But on the particular server (only running one domain) that this was on, this issue has not came back after disabling the rule.
You got a segfault from a rule? That is an absolute first, and would represent a massive unreported bug in modsecurity that no one has ever run into - could you tell us more about this? In all my years with modsecurity (and I have been using it longer than anyone) I've never seen that happen, or ever seen anyone report a rule causing a segfault.

Please tell me more.
ikkk
Forum User
Forum User
Posts: 47
Joined: Wed Jan 05, 2011 3:09 pm

Re: Apache bus error with mod_security, httpd-debuginfo miss

Unread post by ikkk »

Just looked through the IRC logs and it was actually this rule (mentioned in #plesk to Scott as i was complaining about the Rule ID in comments and actual ruleid in the rule being mismatched)

Rule 330005: Blocklist of known malware sites w/ Anti-evasion features but in the actual rule its id:360005

This was causing issues on a http authentication, if you put an incorrect username/password all was fine, but if you put a correct username with any password it would just give you a blank white screen and that bus error would appear in the logs.

It was also happening on a magento installation as well but i cant recall the case surrounding that.

Both of these installs were on OpenVZ setups using the asl-lite rules only. No issues with any beancounters being hit in the openvz setup or such, this is the first thing we ruled out. We also disabled a lot of php modules to check it wasnt things like eaccelerator causing the issue etc.

The error with the http authentication (.htaccess/.passwd in cgi-bin) did not occur straightaway, sometimes it would work for a good few hours after an apache restart, and then it would just go, sometimes an apache restart would sort it, sometimes it wouldnt. I havent heard anything from this client since this rule was disabled so i can only assume all is ok still

Just to clarify - these are not on plesk installs, and were using mod_ruid2 - which was also disabled to see if this was the problem, php was also tried as both php5.2.17 and php5.3.6
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: Apache bus error with mod_security, httpd-debuginfo miss

Unread post by mikeshinn »

Rule 330005: Blocklist of known malware sites w/ Anti-evasion features but in the actual rule its id:360005
Thank you for the reply. So when you ran your backtrace, can you tell me where in the modsecurity code this error is occurring?

If you do not know how to do this, please see this article:

https://www.atomicorp.com/wiki/index.php/Apache

If you didnt use a backtrace, then I have bad news - its probably not fixed. What you have there is a symptom, and you probably just treated the symptom. :-(

So if you did run a backtrace, skip this part, this is just a refresher on segfaults:

The joy of segfaults

Segfaults, as you may know, are memory faults or "bus errors". All odd terms, but it all means the same thing = memory error. Specifically, it means you are trying to access memory the CPU physical can not access. That rule uses a lot of memory, and when you disable it you will use less. As a result, theres more memory to play with. These segfaults generally happen when theres a bug in some code somewhere that isnt using memory correctly. Now the bug is still there, and whatever isn't using memory correctly is still doing that, you just don't "see" it happening because you have more memory to work with and a lower probability of that happening (thats also why people see segfault happen "frequently" but not constantly, or periodically in general - they can happen constantly if you have a REALLY broken application or system.

Think of it like a bucket with holes in it, when its got less water in it, less will leak out, if it has even less nothing leaks out - but if you fill it up those holes start to leak. Its not the water (rules in this case) that cause the water to leak, its the holes (code). So if you have a segfault, its not the rules - they are just the harmless water, something else is causing the system to not use memory correctly. And with apache its all shared, so anything thats not doing that right it punching holes higher up on the bucket. Too much water, and it starts to leak out.

Now, with that said, I can say this with 99% certainty because you aren't the first person to think that modsecurity itself was the cause of a segfault, it seems logical I changed this and it stopped. However, each time we ran a backtrace it turned out to be something else entirely that was the cause. Not the rules and not modsec. Like I said, this isnt something thats ever occurred with modsecurity, it just seems like it is because you are treating the symptom and not the actual cause and that rule uses a lot of memory. Its an "innocent bystander" that has been blammed constantly for segfaults, and every time (look at faris posts for example) when we looked into this, low and behold it was something else.

I'd be delighted to help you with this because hopefully we can finally put this to bed once and for all. If you could run that backtrace I'm sure the entire Apache community would benefit because its a bug somewhere, and it needs to get fixed. Disabling that rule isnt actually fixing your problem, expect it to happen again at some point if something else needs a lot of memory. So far we've found bugs in Apache, APR, several apache modules (but not modsecurity), PHP, accelerators, encoders, lots of htaccess files (with loops in some cases), web applications and even virtualization bugs.

So we would really really love to see that backtrace. :-)
breun
Long Time Forum Regular
Long Time Forum Regular
Posts: 2813
Joined: Sat Aug 20, 2005 9:30 am
Location: The Netherlands

Re: Apache bus error with mod_security, httpd-debuginfo miss

Unread post by breun »

I tried to create a backtrace, but since CentOS doesn't provide an httpd-debuginfo package I couldn't follow the instructions.

We saw Apache segfaulting with mod_security enabled, but no segfaults with mod_security disabled. (I'm not saying mod_security was the culprit, but that's what we saw.) After upgrading to httpd 2.2.17 there were no segfaults anymore, also with mod_security loaded, so there were no backtraces to create. This was on a Virtuozzo guest running CentOS 5.6 32-bit.
Lemonbit Internet Dedicated Server Management
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: Apache bus error with mod_security, httpd-debuginfo miss

Unread post by mikeshinn »

I tried to create a backtrace, but since CentOS doesn't provide an httpd-debuginfo package I couldn't follow the instructions.
Damn annoying that.
After upgrading to httpd 2.2.17 there were no segfaults anymore, also with mod_security loaded
Not surprised, I personally tracked down an APR bug in the 2.2.3 Apache in RHEL 5 and Centos 5 that was causing segfaults on over a dozen different peoples systems, the bug was fixed in 2.2.17. So I know for a fact that there is absolutely a nasty bug in APR that causes segfaults in all 2.2.3 versions of apache in the RHEL/Centos 5 universe.
ikkk
Forum User
Forum User
Posts: 47
Joined: Wed Jan 05, 2011 3:09 pm

Re: Apache bus error with mod_security, httpd-debuginfo miss

Unread post by ikkk »

I have contacted the user to see how things are at present and to see if he is ok with us to do some further testing.
breun
Long Time Forum Regular
Long Time Forum Regular
Posts: 2813
Joined: Sat Aug 20, 2005 9:30 am
Location: The Netherlands

Re: Apache bus error with mod_security, httpd-debuginfo miss

Unread post by breun »

mikeshinn wrote:Not surprised, I personally tracked down an APR bug in the 2.2.3 Apache in RHEL 5 and Centos 5 that was causing segfaults on over a dozen different peoples systems, the bug was fixed in 2.2.17. So I know for a fact that there is absolutely a nasty bug in APR that causes segfaults in all 2.2.3 versions of apache in the RHEL/Centos 5 universe.
Do you know if Red Hat plans on backporting the fix?
Lemonbit Internet Dedicated Server Management
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: Apache bus error with mod_security, httpd-debuginfo miss

Unread post by mikeshinn »

Do you know if Red Hat plans on backporting the fix?
No idea. They havent fixed it, thats for sure.
breun
Long Time Forum Regular
Long Time Forum Regular
Posts: 2813
Joined: Sat Aug 20, 2005 9:30 am
Location: The Netherlands

Re: Apache bus error with mod_security, httpd-debuginfo miss

Unread post by breun »

Have you got the Bugzilla link for this issue?
Lemonbit Internet Dedicated Server Management
Post Reply