The segfaults have gone. All hail the Bus error!

[faris]

Hehe. Very strange. Following a recent kernel update, the apache segfaults (and often associated glibc errors) I've been seeing have been vanquished.

Unfortunately they've been replaced by:

Code: Select all

[notice] child pid 9708 exit signal Bus error (7)

Before anyone suggests it, the system's memory is 100% fine and coincidentally has been recently tested along with all other main hardware components (and in any case is ECC). This is more likely indicating a lack of memory for apache to use somehow, even though the VPS in question has 2Gb allocated to it and normally uses less than 1Gb of that, or is just a different manifestation of the original bug/oddity.

So although the words have changed, the music is still the same

Anyway, the reason for my posting is just to warn people to look out for this as well as for segfaults if you run a Virtuozzo or OpenVZ-based system.

Faris.

[mikeshinn]

Bus Errors are caused by a program crashing, and since this is apache it could be a web app, (CGI, PHP, PERL, etc.), a bug in PHP, an accelerator, etc. - it means the program crashed. A bus error can happen for other reasons, like total incompatibility for example if someone writes a program for one architecture and then runs it on a different one (Intel vs AMD, PPC vs MIPS, etc.). This can also happen with a bug in an accelerator, like PECL or even in PHP in code is cached and fubar somehow. So check there too.

Really all this means is someone is running buggy software on your machine.

[faris]

Interesting...

I don't really know where/how to look for anything that might fit the bill. There are no executables on here that I've not installed myself, no unusual accelerators, and I've not added anything "extra" really. Well, there are a couple of things, but I've tested and neither *seem* to trigger the problem.

Also it is pretty much identical behaviour to the segfaults. After a mod_sec rules update I'll usually get a few (even after I've reduced the two big blacklists), I restart apache a few times, and then it is usually stable for a long time.

I've had more than the usual number after today's rules update, mind you. I'm guessing there have been significant additions since I last updated, which means I'll probably have to reduce the blacklists even more.

I've not seen any segfaults or bus errors on the other two systems we have runnning ASL for a very long time, but both are very lightly loaded compared to this one and both routinely have the blacklists reduced after a rules update.

I may try to provoke one of the other systems over the weekend by not reducing the blacklist to see if I get a bus error or whether I get a segfault.

I'm assuming it isn't a script on one particular system because I get segfaults (or used to get them) on all three. I always used to suspect horde because way back in the past I used to be able to reliably get a segfault after a reboot by logging in to webmail. But not these days.

Faris.

[mikeshinn]

Just remember bus errors are full crashes of an application - it means a web application itself crashed. So you need to look at your apps, and what they use directly. Check your PHP extension order too - and bus errors (segfaults too) can be caused by apps that dont like the version of PHP you have installed for example.

A bus error can also be caused by a bug in PHP or some PHP extension thats not keen to your app can cause it to crash and you would get a bus error. Apache modules don't cause bus errors, its an application problem in the application itself, or its interpreter (like PHP, Perl, etc.) and those libraries. Apaches just in the middle and not the cause, dont waste your time with anything other than tracing down the app thats cause the bus error. Anything else is just treating the symptom.

To that end you should look at the apache debugging modules that use the exception hooks and use them to help you track down your buggy apps:

http://people.apache.org/~trawick/exception_hook.html

[faris]

Unfortunately there are no pre-compiled versions of those modules - and having looked into mod_whatkilledme in the past, trying to get that up and running was not something we could do on our production systems.

The only common thing between all three systems that exhibit this behaviour is the version of Apache, suhosin and Possibly Ioncube but I suspect at least one of them has a different version to the others. Horde webmail is the only common PHP script between the three.

I'll keep plodding away and maybe eventually we'll find the cause.



Faris.