Apache segmentation fault

energylevel · Unread post by **energylevel** » Tue Aug 19, 2008 9:29 am

I keep getting this Ossec alert a lot:

Rule: 30104 fired (level 12) -> "Apache segmentation fault

No idea was exacly it means and what could be the cause, can anyone shed some some light on it for me please?

Thanks .....

Unread post by **scott** » Tue Aug 19, 2008 10:57 am

It means you're getting an Apache Segmentation fault.

energylevel · Unread post by **energylevel** » Tue Aug 19, 2008 11:02 am

What exacly is it? Is it usually due to a resource problem?

Unread post by **scott** » Tue Aug 19, 2008 12:43 pm

Id have to see a core dump really, there are zillions of things that could cause that to happen.

energylevel · Unread post by **energylevel** » Wed Aug 20, 2008 10:09 am

Thnaks Scott, I'm not even sure how to do a core dump, on a Virtuozzo VPS/CentOS4 ??

Unread post by **scott** » Wed Aug 20, 2008 11:06 am

No problem, its up on the wiki:
http://www.atomicorp.com/wiki/index.php/Apache

energylevel · Unread post by **energylevel** » Wed Aug 20, 2008 1:50 pm

Thanks Scott, I followed those instructions, didn't have a debug.conf file present so created one, in my error log I'm seeing a lot of this error:

[Mon Aug 18 16:36:27 2008] [error] [client 89.107.56.253] ModSecurity: Rule processing failed. [hostname "89.107.56.16"] [uri "/"] [unique_id "vAqeR38AAAEAADf6vP8AAAAH"]

Unread post by **scott** » Wed Aug 20, 2008 3:30 pm

Im not on the mod_security team. You're better off sending that to support@atomicorp.com

aus-city · Unread post by **aus-city** » Wed Aug 20, 2008 6:58 pm

They are aware of the issue I have got two servers doing this. I am asking for a patch while its being looked at to detect these 'rule processing failed' in the domain error logs and simply kill httpd, restart it (to get out the dead pids from the rule processing failing) and then asl -s -f

One server I have had to totally disable mod-sec as it brings apache down as its a very busy server.

My advise regularily check your error logs. As soon as you see rule processing failed:

killall httpd
/etc/init.d/httpd start
asl -s -f

Check the logs again. If you see rule processing failed repeat it.

If the rules are failing your killing traffic as they are failing and not processing.

faris · Unread post by **faris** » Fri Aug 29, 2008 12:04 am

We started to get this at 2am this morning (lots of segfaults and the odd "Rule processing failed".

We just moved the VPS in question from one hardware node to another, but that was completed at midnight. Since then I've fiddled around and remember doing an asl -u but ...

It might also have to do with memory. There's a slightly different limitation in place on the new hardware node. It was set to 4Gigs on the old one but seems to be 2Gigs on the new one. I've just upped it to 4Gigs to see what happens.

Unread post by **scott** » Fri Aug 29, 2008 9:50 am

Yeah let me know how that goes, I know where the problem is now, I just don't know what causes it yet. One person did email us to say they figured it out, but then didn't tell us what it was!

faris · Unread post by **faris** » Fri Aug 29, 2008 1:51 pm

Well, I'm completely lost with this one.

The problem is semi-reproducable: with load over a certain (unknown, but low) level, I can get one site to produce the rule errors on demand just by loading the page (under almost no load the problem does not happen).

Interestingly, so far I'm ONLY seeing rule processing errors on one particular site, plus ALL horde webmail usage where just about ever file read results in the rule error.

The segfaults are something else though - I can't put my finger on them. Sometimes they happen immediately after a rule processing error, other times they are apparently stand-alone. Sometimes there's only one, sopmetimes a whole string of them.

It might have something to do with memory, but not in the way I thought. No matter what I do, I cannot get this moved VPS to consume more than 580Mb of RAM, even though it has a 4Gb limit. This compares with the same VPS which was happily gobbling 4Gb when on the old hardware node.

Faris.

Unread post by **scott** » Fri Aug 29, 2008 2:23 pm

If you get some core dumps from it, take a look at them with:

gdb /usr/sbin/httpd core.XXXXX

I see this:
0 0x000067cd764e1b26 in hook_insert_filter (r=0x70774385b78)
at mod_security2.c:945
945 mod_security2.c: No such file or directory.
in mod_security2.c

For the curious, that is losing track of the configuration directives (NOT the rules). I *think* the rule-processing errors are fallout from after this happens. Anyway, someone did claim to isolate the problem. I just havent gotten a response from them about it in a few days

aus-city · Unread post by **aus-city** » Tue Sep 09, 2008 6:01 am

Hi Scott,

Please keep us informed I got a server doing this and I can get a sale for you if it can be fixed.

Can the fix what ever it is be included in an asl update as we all seem to have this devil hanging around.

I believe the busier the server the worse it is. My server gets about 4000 events per hour as logged by asl / ossec level 8s on occasion. This server I may get a few days at best.

I tested mod-sec on a busier server it will go down in hours without question, maybe 2 to 8 but it will go down.

Also Scott if it helps it can do right off after a rule update. Not bad when I do them as I test after an asl -u but the automatic ones are a killer in the middle of the night.

Can automatic rule updates be disabled as this would help until its properly fixed.

Unread post by **scott** » Tue Sep 09, 2008 7:18 am

Yeah they can be disabled, in /etc/asl/config:

AUTOMATIC_UPDATES="daily"

change that to off