Apache segmentation fault

Unread post by **scott** » Thu Oct 02, 2008 2:35 pm

I havent seen a core dump from him to confirm that. Keep in mind that a lot of things will cause apache to segfault, in this case we have established two conditions in relation to mod_security:

1) It will segfault on some systems with *NO* rules. Meaning that blacklists, etc have absolutely nothing to do with this.

2) VPS's will segfault after they initially start. After some time this will cease to occur. This is where turning rules off helped.

We've also caught a lot of segfaults from other things, including php, and mod_python that turning off mod_mem_cache cleared up. So Id say that we're dealing with something a lot more low level in apache and the way DSO's tie into it.

aus-city · Unread post by **aus-city** » Mon Oct 06, 2008 6:34 pm

Scott,

On a full dedicated server I find that despite the mod_cache being off when I find when mod_security has crashed and fails to process rules, even killall httpd and then asl -s -f sometimes immediately after doing this, I check my busiest domain error logs and immediately see rule processing failed (right after you see httpd has been restarted by the ssl certificate message).

Sometimes it takes two or three restarts and away it goes again for a day or few days.

Just wanted to check if you had seen this, sometimes multiple restarts are required.

Also by any chance does mod_security 2.57 have any fixes?

Unread post by **scott** » Tue Oct 07, 2008 9:50 am

We'd really need to see a core dump to get an idea of whats going on there. Check out this page on how to set it up:

http://www.atomicorp.com/wiki/index.php/Apache

dcottle · Unread post by **dcottle** » Tue Oct 07, 2008 9:21 pm

Scott I will dump tonight.

dcottle · Unread post by **dcottle** » Wed Oct 08, 2008 7:27 am

Hi Scott,

I can't get Fedora to dump with those instructions. I kill -11 PID of httpd and nothing. Found some other doc but it then dumped 0 byte files.

I found details on redhats page and it works, got to edit 5 files but it dumps. I would post the link but my IP is locked out of atomic Michael should be aware I emailed him he fixed it last time.

Anyway as soon as it happens I will have some dumps for you both F8 and F9.
How do I send them they are large when it happens?

faris · Unread post by **faris** » Wed Oct 08, 2008 8:50 am

This isn't my area, but really what should happen is that you will get a core dump the moment apache segfaults. No user intervention is required. All you need is to put the dump path in httpd.conf and restart apache and that's it.

Scott has also mentioned to me in the past that it is helpful to have httpd-debuginfo installed in order to get more useful info in the core dump.
You might want to try installing that first (backup first!)

Faris.

faris · Unread post by **faris** » Wed Oct 08, 2008 8:53 am

Oh, one more thing. If your system is prone to long spates of segfaults (e.g. scores in a row), then keep in mind that the core dumps can be huge (200Mb each in our case).

When we were initially looking into this, the server suddenly went unresponsive. The culprit was apache dumping the core a few zillion times, filling up the disk and of course slowing everything down.

Faris.

Unread post by **scott** » Wed Oct 08, 2008 9:57 am

yeah you don't force it to core, that won't get you what you want here. You need to set it up to do it, and then wait for it to segfault. Once you have a core, we just need the backtrace from gdb.

aus-city · Unread post by **aus-city** » Wed Oct 08, 2008 6:46 pm

No problem I just did the kill -11 to test I would see actual dumps. Fedora and Redhat I found have dumps disabled by default it takes a bit of file editing:

http://kbase.redhat.com/faq/FAQ_80_3652.shtm

I found three dumps today, I just emailed you the URLs for them.

Still waiting for the actual rule processing to fail. Should I put back on mod-cache to entice it or we wait patiently?

Unread post by **scott** » Wed Oct 08, 2008 6:48 pm

No dont enable mod_mem_cache, we've already established that it will cause problems.

dcottle · Unread post by **dcottle** » Thu Oct 09, 2008 1:55 am

No problems! I just installed httpd-debuginfo to both servers and restarted httpd so we just wait for some dumps.

Unread post by **scott** » Thu Oct 09, 2008 9:29 am

To force it, I would fire off several instances of ab (apache benchmark) from other servers at once.

dcottle · Unread post by **dcottle** » Tue Oct 14, 2008 7:10 pm

Typical ever since I got httpd-debuginfo and coredump it's behaved itself almost like it's got a will not to break. Tried benchmarks it just keeps going and it's had 3 rule updates.

Well sooner or later mod-security will break and 'got you'.

aus-city · Unread post by **aus-city** » Sun Oct 19, 2008 2:48 pm

Its STILL behaving

Can running httpd with core dumps changed something memory related that fixes this, it can't be co-incidence?

faris · Unread post by **faris** » Sun Oct 19, 2008 3:06 pm

Most likely it is the httpd-debuginfo that's changed things and not having core dumps enabled.

I regret that I don't have time to test this myself in order to find out if you are just plain lucky or if it is likely to solve the issue more generally. (though at the end of the day, there is a bug somewhere - be it in Apache or mod_sec or elsewhere).

Personally I'm just relieved that you have a "fix" for your problems.