Debugging core dump from Apache segfault

[chrismcb]

Hi,

I've been getting segfaults from apache and have core dumps to analyse.

They end with:

Code: Select all

Core was generated by `/usr/sbin/httpd'.
Program terminated with signal 11, Segmentation fault.
#0  0x000078a901078976 in execute () from /etc/httpd/modules/libphp5.so

From searching online, I should include PHP for the search, however it shows:

Code: Select all

Reading symbols from /usr/bin/php...(no debugging symbols found)...done.

My installed php-related versions are:

Code: Select all

# rpm -qa | grep 'php'
psa-appvault-phpmoney-1.3-8204320080409011649
psa-appvault-phpdig-1.85-8203120080409011645
php-5.2.17-1.el5.art
php-mcrypt-5.2.17-1.el5.art
asl-php-process-5.4.6-4.el5.art
psa-appvault-phpbugtracker-1.19-8203820080416050605
php-common-5.2.17-1.el5.art
php-mysql-5.2.17-1.el5.art
psa-php5-configurator-1.6.1-cos5.build1013120126.11
php-gd-5.2.17-1.el5.art
asl-php-pdo-5.4.6-4.el5.art
psa-appvault-phpsurveyor-0.98-8204320080409011723
psa-appvault-phpbook-1.50-8203220080409011638
psa-appvault-phpwebsite-0.10.2-8203420080409011738
psa-appvault-phpwiki-1.3.11-8204320080409011808
php-ioncube-loader-4.0.5-1.el5.art
php-pdo-5.2.17-1.el5.art
php-imap-5.2.17-1.el5.art
asl-php-common-5.4.6-4.el5.art
asl-php-gd-5.4.6-4.el5.art
psa-appvault-phpmyvisites-2.3-8202820080409011701
php-mbstring-5.2.17-1.el5.art
asl-php-5.4.6-4.el5.art
psa-appvault-phpads-2.0.8-8203520080409011611
psa-appvault-phpmyfamily-1.4.1-8203420080409011655
php-pear-1.7.2-2.el5.art
php-devel-5.2.17-1.el5.art
asl-php-mysql-5.4.6-4.el5.art
psa-appvault-phprojekt-5.2-8200820080409011713
roadsend-php-libs-2.9.8-9.el5.art
php-cli-5.2.17-1.el5.art
php-xml-5.2.17-1.el5.art
asl-php-cli-5.4.6-4.el5.art

Trying to install debuginfo for php results in:

Code: Select all

# debuginfo-install php
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * 4PLESK: plesk.ro.4psa.com
 * 4PSA: download1.ro.4psa.com
 * atomic: www7.atomicorp.com
 * base: mirror.sov.uk.goscomb.net
 * extras: mirror.sov.uk.goscomb.net
 * updates: mirror.sov.uk.goscomb.net
4PLESK                                                                                                                                                                                                                |  951 B     00:02
4PSA                                                                                                                                                                                                                  | 1.1 kB     00:00
asl-3.0                                                                                                                                                                                                               | 1.9 kB     00:00
atomic                                                                                                                                                                                                                | 1.9 kB     00:00
base                                                                                                                                                                                                                  | 1.1 kB     00:00
extras                                                                                                                                                                                                                | 2.1 kB     00:00
mod-pagespeed                                                                                                                                                                                                         |  951 B     00:00
serverdensity                                                                                                                                                                                                         |  951 B     00:00
updates                                                                                                                                                                                                               | 1.9 kB     00:00
Checking for new repos for mirrors
Could not find debuginfo for main pkg: php-5.2.17-1.el5.art.x86_64
Could not find debuginfo pkg for dependency package bzip2-libs-1.0.3-6.el5_5.x86_64
Could not find debuginfo pkg for dependency package glibc-2.5-81.el5_8.4.x86_64
Could not find debuginfo pkg for dependency package glibc-2.5-81.el5_8.4.x86_64
Could not find debuginfo pkg for dependency package glibc-2.5-81.el5_8.4.x86_64
Could not find debuginfo pkg for dependency package glibc-2.5-81.el5_8.4.x86_64
Could not find debuginfo pkg for dependency package glibc-2.5-81.el5_8.4.x86_64
Could not find debuginfo pkg for dependency package e2fsprogs-libs-1.39-34.el5_8.1.x86_64
Could not find debuginfo pkg for dependency package glibc-2.5-81.el5_8.4.x86_64
Could not find debuginfo pkg for dependency package glibc-2.5-81.el5_8.4.x86_64
Could not find debuginfo pkg for dependency package openssl-0.9.8e-22.el5_8.4.x86_64
Could not find debuginfo pkg for dependency package glibc-2.5-81.el5_8.4.x86_64
Could not find debuginfo pkg for dependency package glibc-2.5-81.el5_8.4.x86_64
Could not find debuginfo pkg for dependency package gmp-4.1.4-10.el5.x86_64
Could not find debuginfo pkg for dependency package krb5-libs-1.6.1-70.el5.x86_64
Could not find debuginfo pkg for dependency package krb5-libs-1.6.1-70.el5.x86_64
Could not find debuginfo pkg for dependency package krb5-libs-1.6.1-70.el5.x86_64
Could not find debuginfo pkg for dependency package glibc-2.5-81.el5_8.4.x86_64
Could not find debuginfo pkg for dependency package glibc-2.5-81.el5_8.4.x86_64
Could not find debuginfo pkg for dependency package glibc-2.5-81.el5_8.4.x86_64
Could not find debuginfo pkg for dependency package glibc-2.5-81.el5_8.4.x86_64
Could not find debuginfo pkg for dependency package glibc-2.5-81.el5_8.4.x86_64
Could not find debuginfo pkg for dependency package openssl-0.9.8e-22.el5_8.4.x86_64
Could not find debuginfo pkg for dependency package libxml2-2.6.26-2.1.15.el5_8.2.x86_64
Could not find debuginfo pkg for dependency package zlib-1.2.3-4.el5.x86_64
No debuginfo packages available to install

Anyone able to offer any advice on getting to the bottom of the trace to figure out which PHP script is causing the segfaults?



Thanks

[faris]

Um...looks like you are running php 5.2.x. I would hazard a guess that that's your problem. No updates, no support, lots of bugs and no fixes for them.

I know a few people have been stuck on php 5.2 in the past for various reasons, but generally they can be overcome, so that's the direction I'd be looking right now if at all possible.

[chrismcb]

Yeah, I know it's a bit dated, but there's a few sites on the platform which are not compatible with 5.3+.

As they are client sites, it's not feasible to redevelop code to upgrade them free of charge.

If only I could keep the sites in question as 5.2 and the rest up to date!


Am I right in saying all I need is php-debuginfo for 5.2.17-1.el5.art?

If not, can you point me in the right direction?

[faris]

There doesn't seem to be one in the repo (at all - not even for 5.3.x).

But what I was getting at was that even if you find a debuginfo rpm and trace the problem, there's not a lot that can be done if you are stuck on 5.2.

[chrismcb]

Thanks Faris, but I'm guessing it's a module or a script that's causing the issue.

With the PHP debugging info, it should be able to pinpoint exactly what is causing the error for me to take further action.

[chrismcb]

There doesn't seem to be one in the repo (at all - not even for 5.3.x).

Question then is, is there a different way to get debuginfo for the Atomic repository for any build?
http://www6.atomicorp.com/channels/atom ... 6_64/RPMS/

I have tried using an alternate build I found online (php-debuginfo-5.2.17-1.el5.vmbleed.x86_64.rpm), but gdb complains with messages like:

Code: Select all

Reading symbols from /etc/httpd/modules/libphp5.so...
warning: the debug information found in "/usr/lib/debug/usr/lib64/httpd/modules/libphp5.so.debug" does not match "/etc/httpd/modules/libphp5.so" (CRC mismatch).

Which was expected anyway.

[scott]

you'll need to change the debug nil line in the spec file if I recall correctly.

[chrismcb]

you'll need to change the debug nil line in the spec file if I recall correctly.

Sorry, can you point me in the right direction with this?

[chrismcb]

Reviving this old thread as I have upgraded to the latest version of PHP:

Code: Select all

php-5.3.17-10.el5.art

But, as faris mentioned, there is no debug packages in the Atomicorp repository for PHP.

Can these be added? or is there another way I can pinpoint the cause of these intermittent segfaults?


Thanks