All PHP/MySql websites are down - Segmentation fault (11)

[damon]

After my yum updates yesterday everything seemed normal, but this morning I started getting calls about webmail not working.

While I was investigating that (Horde always seems to be a problem) I found a ton of these in /var/log/httpd/error_log

Code: Select all

[Fri Mar 12 14:37:05 2010] [notice] child pid 5267 exit signal Segmentation fault (11)
[Fri Mar 12 14:37:07 2010] [notice] child pid 5260 exit signal Segmentation fault (11)
[Fri Mar 12 14:37:10 2010] [notice] child pid 5274 exit signal Segmentation fault (11)
[Fri Mar 12 14:37:13 2010] [notice] child pid 5254 exit signal Segmentation fault (11)
[Fri Mar 12 14:37:13 2010] [notice] child pid 5278 exit signal Segmentation fault (11)
[Fri Mar 12 14:37:15 2010] [notice] child pid 5277 exit signal Segmentation fault (11)

Not very descriptive, i've done a lot of googling but this seems to have many meanings.

This is a fully updated Centos/ASL system.

Regular html sites are working and httpd and mysqld services are running, but all sites that use php/mysql are blank pages.

The /var/log/messages log is getting big with over 270,000 lines, here are the latest entries

Code: Select all

Mar 12 14:44:45 memwow kernel: grsec: From 207.166.3.62: signal 11 sent to /usr/sbin/httpd[httpd:6058] uid/euid:48/48 gid/egid:48/48, parent /usr/sbin/httpd[httpd:5221] uid/euid:0/0 gid/egid:0/0
Mar 12 14:44:45 memwow kernel: grsec: From 207.166.3.62: signal 11 sent to /usr/sbin/httpd[httpd:6058] uid/euid:48/48 gid/egid:48/48, parent /usr/sbin/httpd[httpd:5221] uid/euid:0/0 gid/egid:0/0
Mar 12 14:44:45 memwow kernel: grsec: From 207.166.3.62: signal 11 sent to /usr/sbin/httpd[httpd:6068] uid/euid:48/48 gid/egid:48/48, parent /usr/sbin/httpd[httpd:5221] uid/euid:0/0 gid/egid:0/0
Mar 12 14:44:45 memwow kernel: grsec: From 207.166.3.62: signal 11 sent to /usr/sbin/httpd[httpd:6068] uid/euid:48/48 gid/egid:48/48, parent /usr/sbin/httpd[httpd:5221] uid/euid:0/0 gid/egid:0/0
Mar 12 14:44:45 memwow kernel: grsec: From 207.166.3.62: signal 11 sent to /usr/sbin/httpd[httpd:6068] uid/euid:48/48 gid/egid:48/48, parent /usr/sbin/httpd[httpd:5221] uid/euid:0/0 gid/egid:0/0
Mar 12 14:44:45 memwow kernel: grsec: From 207.166.3.62: signal 11 sent to /usr/sbin/httpd[httpd:6058] uid/euid:48/48 gid/egid:48/48, parent /usr/sbin/httpd[httpd:5221] uid/euid:0/0 gid/egid:0/0
Mar 12 14:44:45 memwow kernel: grsec: From 207.166.3.62: signal 11 sent to /usr/sbin/httpd[httpd:6068] uid/euid:48/48 gid/egid:48/48, parent /usr/sbin/httpd[httpd:5221] uid/euid:0/0 gid/egid:0/0
Mar 12 14:44:45 memwow kernel: grsec: From 207.166.3.62: signal 11 sent to /usr/sbin/httpd[httpd:6058] uid/euid:48/48 gid/egid:48/48, parent /usr/sbin/httpd[httpd:5221] uid/euid:0/0 gid/egid:0/0
Mar 12 14:44:45 memwow kernel: grsec: From 207.166.3.62: signal 11 sent to /usr/sbin/httpd[httpd:6068] uid/euid:48/48 gid/egid:48/48, parent /usr/sbin/httpd[httpd:5221] uid/euid:0/0 gid/egid:0/0
Mar 12 14:44:45 memwow kernel: grsec: From 207.166.3.62: signal 11 sent to /usr/sbin/httpd[httpd:6068] uid/euid:48/48 gid/egid:48/48, parent /usr/sbin/httpd[httpd:5221] uid/euid:0/0 gid/egid:0/0
Mar 12 14:44:45 memwow kernel: grsec: more alerts, logging disabled for 10 seconds
Mar 12 14:44:45 memwow kernel: grsec: more alerts, logging disabled for 10 seconds
Mar 12 14:44:50 memwow xinetd[2709]: START: smtp pid=6087 from=69.87.62.171
Mar 12 14:44:50 memwow xinetd[2709]: START: smtp pid=6087 from=69.87.62.171
Mar 12 14:44:51 memwow xinetd[2709]: START: smtp pid=6088 from=69.87.62.176
Mar 12 14:44:51 memwow xinetd[2709]: START: smtp pid=6088 from=69.87.62.176
Mar 12 14:44:52 memwow xinetd[2709]: EXIT: smtp status=0 pid=6088 duration=1(sec)
Mar 12 14:44:52 memwow xinetd[2709]: EXIT: smtp status=0 pid=6088 duration=1(sec)
Mar 12 14:44:52 memwow xinetd[2709]: EXIT: smtp status=0 pid=6087 duration=2(sec)
Mar 12 14:44:52 memwow xinetd[2709]: EXIT: smtp status=0 pid=6087 duration=2(sec)

[stormlord]

I have exact the same problème
nothing is working exept http basic site.
Email not working too.
I have try to boot different kernel , ...Nothing
Remove et install asl , ossec , etc.. , ...Nothing

don't know what to try now. I will open ticket

12Mar 16:02:57 12 30104 [notice] child pid 8758 exit signal Segmentation fault (11)
12Mar 16:02:42 12 30104 [notice] child pid 8757 exit signal Segmentation fault (11)
12Mar 16:02:37 12 30104 [notice] child pid 8752 exit signal Segmentation fault (11)
12Mar 16:02:37 12 30104 [notice] child pid 8677 exit signal Segmentation fault (11)
12Mar 16:02:37 12 30104 [notice] child pid 8676 exit signal Segmentation fault (11)
12Mar 16:02:37 12 30104 [notice] child pid 8674 exit signal Segmentation fault (11)
12Mar 16:02:37 12 30104 [notice] child pid 8675 exit signal Segmentation fault (11)
12Mar 16:02:32 12 30104 [notice] child pid 8666 exit signal Segmentation fault (11)
12Mar 16:02:32 12 30104 [notice] child pid 8665 exit signal Segmentation fault (11)
12Mar 16:02:32 12 30104 [notice] child pid 8645 exit signal Segmentation fault (11)
12Mar 16:02:32 12 30104 [notice] child pid 8623 exit signal Segmentation fault (11)
12Mar 16:02:17 12 30104 [notice] child pid 8568 exit signal Segmentation fault (11)
12Mar 16:02:17 12 30104 [notice] child pid 8567 exit signal Segmentation fault (11)
12Mar 16:02:12 12 30104 [notice] child pid 8566 exit signal Segmentation fault (11)
12Mar 16:02:12 12 30104 [notice] child pid 8565 exit signal Segmentation fault (11)
12Mar 16:02:02 12 30104 [notice] child pid 8554 exit signal Segmentation fault (11)
12Mar 16:02:02 12 30104 [notice] child pid 8555 exit signal Segmentation fault (11)
12Mar 16:02:02 12 30104 [notice] child pid 8532 exit signal Segmentation fault (11)
12Mar 16:02:02 12 30104 [notice] child pid 8466 exit signal Segmentation fault (11)
12Mar 16:01:52 12 30104 [notice] child pid 8382 exit signal Segmentation fault (11)
12Mar 16:01:52 12 30104 [notice] child pid 8381 exit signal Segmentation fault (11)
12Mar 16:01:52 12 30104 [notice] child pid 8379 exit signal Segmentation fault (11)
12Mar 16:01:52 12 30104 [notice] child pid 8378 exit signal Segmentation fault (11)
12Mar 16:01:42 12 30104 [notice] child pid 8373 exit signal Segmentation fault (11)
12Mar 16:01:42 12 30104 [notice] child pid 8372 exit signal Segmentation fault (11)
12Mar 16:01:42 12 30104 [notice] child pid 8348 exit signal Segmentation fault (11)
12Mar 16:01:42 12 30104 [notice] child pid 7714 exit signal Segmentation fault (11)
12Mar 16:01:32 12 30104 [notice] child pid 7713 exit signal Segmentation fault (11)
12Mar 16:01:32 12 30104 [notice] child pid 7705 exit signal Segmentation fault (11)
12Mar 16:01:32 12 30104 [notice] child pid 7700 exit signal Segmentation fault (11)
12Mar 16:01:32 12 30104 [notice] child pid 7156 exit signal Segmentation fault (11)
12Mar 16:01:17 12 30104 [notice] child pid 7646 exit signal Segmentation fault (11)
12Mar 16:01:17 12 30104 [notice] child pid 7194 exit signal Segmentation fault (11)
12Mar 16:01:17 12 30104 [notice] child pid 7193 exit signal Segmentation fault (11)
12Mar 16:01:17 12 30104 [notice] child pid 7174 exit signal Segmentation fault (11)
12Mar 16:01:07 12 30104 [notice] child pid 7154 exit signal Segmentation fault (11)
12Mar 16:01:07 12 30104 [notice] child pid 7153 exit signal Segmentation fault (11)
12Mar 16:01:07 12 30104 [notice] child pid 7152 exit signal Segmentation fault (11)
12Mar 16:01:02 12 30104 [notice] child pid 7139 exit signal Segmentation fault (11)

[damon]

So it looks like I have company.

Did you do a Yum update recently?

[stormlord]

yum update yesterday like you.
My customers call too, not very happy.

[damon]

My customers call too, not very happy.

I hear that! It's a helpless feeling when all the sites go down at once.

Sorry, but I'm glad someone else has the same problem, I was about to give 1and1 some grief about possible hardware errors because of what I've read about segmentation faults caused by memory errors.

But we probably both didn't develop the same hardware problem at the same time.

[damon]

This crash is different from anything I've seen before, because lots of things still work on the server.

email/imap
plain html sites
plesk panel

I've got phpMyAdmin installed on one domain with access to all the databases on the server, that works fine.

But domains running web applications will load. big problem.

[stormlord]

Its not an hardware probleme I run my server on my vcenter.
The problme come after the last update for sure.

[BruceLee]

try do disable/remove modules. e.g. ioncube,zend,eaccelerator and see if the error persists.
are you running latest versions?
check with yum check-update

[damon]

try do disable/remove modules. e.g. ioncube,zend,eaccelerator and see if the error persists.
are you running latest versions?
check with yum check-update

Two days ago I did the update that I asked about in this thread
http://www.atomicorp.com/forum/viewtopic.php?f=3&t=3949

Yesterday (Mar 11) I ran yum update again and found a new version of eaccelerator from atomic repos (previous was a release candidate) and installed it.

Just now I ran the check-update and found yet another eaccelerator and installed it.

Didn't have any affect.

Everything LOOKS ok with PHP and modules looking at phpinfo and php -v at command line gives this

PHP 5.2.13 (cli) (built: Mar 2 2010 18:08:48)
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2010 Zend Technologies
with eAccelerator v0.9.6, Copyright (c) 2004-2010 eAccelerator, by eAccelerator
with the ionCube PHP Loader v3.3.11, Copyright (c) 2002-2010, by ionCube Ltd., and
with Zend Optimizer v3.3.9, Copyright (c) 1998-2009, by Zend Technologies



Also tried disabling all php modules by renaming the php.d directory where they are listed and restarted httpd. That gives this when command php -v

PHP 5.2.13 (cli) (built: Mar 2 2010 18:08:48)
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2010 Zend Technologies

Still no php websites will load

[diego]

I have the same problem, it was solved when i removed php-eaccelerator

use: yum remove php-eaccelerator

to remove it, and then run:

service httpd restart

and all the sites are back again.

The problem here is with php-eaccelerator and php 5.2.13

Code: Select all

php -v
PHP 5.2.13 (cli) (built: Mar  2 2010 18:08:48) 
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2010 Zend Technologies
    with the ionCube PHP Loader v3.3.11, Copyright (c) 2002-2010, by ionCube Ltd., and
    with Zend Optimizer v3.3.9, Copyright (c) 1998-2009, by Zend Technologies
    with Suhosin v0.9.29, Copyright (c) 2007, by SektionEins GmbH

[damon]

I have the same problem, it was solved when i removed php-eaccelerator
use: yum remove php-eaccelerator
to remove it, and then run:
service httpd restart
and all the sites are back again.

Trying anything at this point.

Yes, that worked for me, running without php-eaccelerator got the sites back up.

But now my question is, how did removing it via yum have a different affect than simply not loading it?

Hopfully Atomic will release a version that works. I'm going to need to update PHP to 5.3 soon, can't wait for the problems that will bring.

[damon]

This might be a clue, I found on the eaccelerator website when I did a search for php 5.2.13

http://eaccelerator.net/ticket/409

So maybe the problems were caused by not clearing the cache? I've never had to do that before, if it's required you would think it would be part of the upgrade.

[BruceLee]

how did I update two days ago:
stop httpd
kill all httpd processes
kill all php-cgi processes
delete all eaccelerator cache files
make copy of eaccelerator.ini
run update
check eaccelerator.ini
start httpd

maybe this procedure is helping you. I had to kill the processes due to the segfaults that caused them to stay "alive".
Maybe enabling debug in eaccelerator helps.

EDIT: forgot to mention that I also killed all left over semaphore arrays:

Code: Select all

ipcs -s | awk ' $3 == "apache" {print $2, $3}' | awk '{ print $1}' | while read i; do ipcrm sem $i; done

[/color]

[Bierteufel]

Last Night i made some elaboration test on my systems:

VPS
CENT OS 5.4
PHP 5.2.13
MYSQL Client API version 5.0.90
Client API header version 5.0.77

with eAccelerator v0.9.6-rc1 the system works.

With later eAccelerator version the system crahes.

"PHP crahed on opline XX ......"

[breun]

After updating php-eaccelerator.i386 from 1:5.2.13_0.9.6-1.el4.art to 1:5.2.13_0.9.6-3.el4.art I also see Apache starting to segfault after a while.