Page 10 of 11
Re:
Posted: Sat Feb 28, 2009 11:51 am
by faris
aus-city wrote:
You put the script into your rc.local so its run on bootup.
Faris's dumbo question of the day - does this need to be run as a background task when started via rc.local ?
i.e.
/path/to/scriptname &
or
perl /path/to/scriptname &
or
what?
I presume not, and that anything in rc.local is just run as an independant process so it can just be /path/to/scriptname
Faris.
Re: Apache segmentation fault
Posted: Sat Feb 28, 2009 12:03 pm
by scott
It doesnt matter really, rc.local runs last so everything else has started already at that point
Re: Apache segmentation fault
Posted: Sun Mar 01, 2009 9:02 am
by faris
Interesting. Thanks.
Incidentally I've reverted to the full ruleset over the weekend. So far memory usage is up (as expected), semaphores are up (as expected) but things are stable and no segfaults.
I'll keep an eye on things on Monday, when the load will get to normal levels, and see what happens.
I've got that script running on one VPS to see if it makes any difference if there's a problem.
Faris.
Re: Apache segmentation fault
Posted: Sun Mar 01, 2009 1:10 pm
by faris
I just tested and it kinda does need an & at the end otherwise you end up with s99 and rc 3 showing up in the processlist permanently.
I guess it doesn't matter though.
Faris.
Re: Apache segmentation fault
Posted: Mon Mar 02, 2009 12:00 pm
by faris
Well, with the full ruleset active, no segfaults. No problems.
So far!
Please cross your fingers that all is now well again.
Faris.
Re: Apache segmentation fault
Posted: Tue Mar 03, 2009 12:29 pm
by faris
Errr...
I thought I'd test the restart script.....so I added an approrpriate error to the error_log
Unfortunately it didn't really work!
Here's what happened. Looks like the semaphore cleaning didn't do its thing or something?
Code: Select all
[Tue Mar 03 16:24:31 2009] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Tue Mar 03 16:24:31 2009] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Tue Mar 03 16:24:31 2009] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Tue Mar 03 16:24:31 2009] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Tue Mar 03 16:24:31 2009] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Tue Mar 03 16:24:31 2009] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Tue Mar 03 16:24:31 2009] [alert] Child 32051 returned a Fatal error... Apache is exiting!
[Tue Mar 03 16:24:31 2009] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Tue Mar 03 16:24:38 2009] [emerg] (22)Invalid argument: couldn't release the accept mutex
And unfortunately apache wasn't restarted. Obviously psmon would have taken care of it if I hadn't done so manually.
Indeed, I'm wondering if psmon got in the way - if psmon restarted apache before the the semaphores were cleaned or something?
EDIT:
Yeah, I think it was psmon. I got this at the same time
Code: Select all
Command executed: /sbin/service httpd restart Exit value: 1 Signal number: 0 Dumped core?: 0
Stopping httpd: [FAILED] Starting httpd: [FAILED]
Faris.
Re: Apache segmentation fault
Posted: Tue Mar 03, 2009 12:47 pm
by faris
Ah. Found the REAL problem!
(and similarly for start)
Should be httpd not apache
Nevertheless, stopping psmon seems liek a good idea.
Here's my revised code:
Code: Select all
sub act {
# stop psmon and give it a chance to stop
system "/etc/init.d/psmon stop";
sleep 2;
# stop apache
#system "killall -9 httpd";
system "/etc/init.d/httpd stop";
# clear semaphores for apache
system "ipcs -s | grep $apacheuser | awk ' { print $2 } ' | xargs ipcrm sem";
#start apache clean
system "/etc/init.d/httpd start";
#give it some time to start, just in case
sleep 10;
#restart psmon now
system "/etc/init.d/psmon start";
# email admin
system "echo 'Service httpd restarted due to processing error at " . scalar(localtime) . "' | mail $adminemail -s $subjectemail";
# don't restart again sooner than 5 minutes, in case of mad apache
sleep 300;
}
EDIT ... ok ok. So what I had was old code. But I just downloaded it. I thought it had been fixed and re-uploaded?
Anyway, it works.
But I note that I'm getting
Code: Select all
Use of uninitialized value in concatenation (.) or string at restart_apache.pl line 68, <GEN0> line 1.
usage: ipcrm [ [-q msqid] [-m shmid] [-s semid]
[-Q msgkey] [-M shmkey] [-S semkey] ... ]
deprecated usage: ipcrm {shm | msg | sem} id ...
Changing sem to -s doesn't work for some reason though. I'm missing something.
Re: Apache segmentation fault
Posted: Tue Mar 03, 2009 2:06 pm
by hostingguy
Try this
Code: Select all
system "for i in `ipcs -s | grep $apacheuser | awk '{print $2}'` ; do ipcrm -s $i; done";
Re: Apache segmentation fault
Posted: Tue Mar 03, 2009 2:25 pm
by faris
Lovely. Thanks.
That makes more sense too.
Faris.
Re: Apache segmentation fault
Posted: Tue Mar 03, 2009 5:15 pm
by faris
Well, even that didn't work. Perl was complaining about potential syntax errors.
There's nothing wrong with the syntax though..
I've had to put the apache stop/start etc and semaphore cleaning in a bash script, and call that from the perl script.
It is quite happy with that
Faris.
Re: Apache segmentation fault
Posted: Sat Mar 07, 2009 11:14 am
by sebas
Ok I managed to have dumps, I also installed -debuginfo, but when I do use gdb I get:
gdb core.22310
GNU gdb Fedora (6.8-27.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <
http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
"/disk1/apache2-gdb-dump/core.22310": not in executable format: File format not recognized
It might be important to say that I installed httpd-debuinfo once I had the dump.
Thanks for your help.
Re: Apache segmentation fault
Posted: Sat Mar 07, 2009 1:25 pm
by scott
Youd do:
gdb /usr/sbin/httpd <path to core>
Re: Apache segmentation fault
Posted: Tue Mar 10, 2009 3:01 pm
by sebas
Ok here is what I get:
This GDB was configured as "x86_64-redhat-linux-gnu"...
warning: Can't read pathname for load map: Input/output error.
Loaded symbols for /usr/sbin/httpd
Reading symbols from /lib64/libm.so.6...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libpcre.so.0...done.
Loaded symbols for /lib64/libpcre.so.0
Reading symbols from /lib64/libselinux.so.1...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /usr/lib64/libaprutil-1.so.0...done.
Loaded symbols for /usr/lib64/libaprutil-1.so.0
Reading symbols from /lib64/libcrypt.so.1...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /usr/lib64/libldap-2.3.so.0...done.
Loaded symbols for /usr/lib64/libldap-2.3.so.0
Reading symbols from /usr/lib64/liblber-2.3.so.0...done.
Loaded symbols for /usr/lib64/liblber-2.3.so.0
Reading symbols from /lib64/libdb-4.3.so...done.
Loaded symbols for /lib64/libdb-4.3.so
Reading symbols from /lib64/libexpat.so.0...done.
Loaded symbols for /lib64/libexpat.so.0
Reading symbols from /usr/lib64/libapr-1.so.0...done.
Loaded symbols for /usr/lib64/libapr-1.so.0
Reading symbols from /lib64/libpthread.so.0...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/libsepol.so.1...done.
Loaded symbols for /lib64/libsepol.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libuuid.so.1...done.
Loaded symbols for /lib64/libuuid.so.1
Reading symbols from /usr/lib64/libpq.so.4...done.
Loaded symbols for /usr/lib64/libpq.so.4
Reading symbols from /usr/lib64/libsqlite3.so.0...done.
Loaded symbols for /usr/lib64/libsqlite3.so.0
Reading symbols from /lib64/libresolv.so.2...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /usr/lib64/libsasl2.so.2...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /lib64/libssl.so.6...done.
Loaded symbols for /lib64/libssl.so.6
Reading symbols from /lib64/libcrypto.so.6...done.
Loaded symbols for /lib64/libcrypto.so.6
Reading symbols from /usr/lib64/libkrb5.so.3...done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /lib64/libnsl.so.1...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libcom_err.so.2...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libk5crypto.so.3...done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /usr/lib64/libz.so.1...done.
Loaded symbols for /usr/lib64/libz.so.1
Reading symbols from /usr/lib64/libkrb5support.so.0...done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libnss_files.so.2...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /lib64/librt.so.1...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /usr/lib64/httpd/modules/mod_auth_basic.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_auth_basic.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_auth_basic.so
Reading symbols from /usr/lib64/httpd/modules/mod_auth_digest.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_auth_digest.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_auth_digest.so
Reading symbols from /usr/lib64/httpd/modules/mod_authn_file.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_authn_file.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_authn_file.so
Reading symbols from /usr/lib64/httpd/modules/mod_authn_alias.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_authn_alias.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_authn_alias.so
Reading symbols from /usr/lib64/httpd/modules/mod_authn_anon.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_authn_anon.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_authn_anon.so
Reading symbols from /usr/lib64/httpd/modules/mod_authn_dbm.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_authn_dbm.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_authn_dbm.so
Reading symbols from /usr/lib64/httpd/modules/mod_authn_default.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_authn_default.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_authn_default.so
Reading symbols from /usr/lib64/httpd/modules/mod_authz_host.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_authz_host.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_authz_host.so
Reading symbols from /usr/lib64/httpd/modules/mod_authz_user.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_authz_user.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_authz_user.so
Reading symbols from /usr/lib64/httpd/modules/mod_authz_owner.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_authz_owner.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_authz_owner.so
Reading symbols from /usr/lib64/httpd/modules/mod_authz_groupfile.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_authz_groupfile.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_authz_groupfile.so
Reading symbols from /usr/lib64/httpd/modules/mod_authz_dbm.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_authz_dbm.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_authz_dbm.so
Reading symbols from /usr/lib64/httpd/modules/mod_authz_default.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_authz_default.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_authz_default.so
Reading symbols from /usr/lib64/httpd/modules/mod_ldap.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_ldap.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_ldap.so
Reading symbols from /usr/lib64/httpd/modules/mod_authnz_ldap.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_authnz_ldap.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_authnz_ldap.so
Reading symbols from /usr/lib64/httpd/modules/mod_include.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_include.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_include.so
Reading symbols from /usr/lib64/httpd/modules/mod_log_config.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_log_config.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_log_config.so
Reading symbols from /usr/lib64/httpd/modules/mod_logio.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_logio.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_logio.so
Reading symbols from /usr/lib64/httpd/modules/mod_env.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_env.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_env.so
Reading symbols from /usr/lib64/httpd/modules/mod_ext_filter.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_ext_filter.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_ext_filter.so
Reading symbols from /usr/lib64/httpd/modules/mod_mime_magic.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_mime_magic.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_mime_magic.so
Reading symbols from /usr/lib64/httpd/modules/mod_expires.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_expires.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_expires.so
Reading symbols from /usr/lib64/httpd/modules/mod_deflate.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_deflate.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_deflate.so
Reading symbols from /usr/lib64/httpd/modules/mod_headers.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_headers.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_headers.so
Reading symbols from /usr/lib64/httpd/modules/mod_usertrack.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_usertrack.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_usertrack.so
Reading symbols from /usr/lib64/httpd/modules/mod_setenvif.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_setenvif.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_setenvif.so
Reading symbols from /usr/lib64/httpd/modules/mod_mime.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_mime.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_mime.so
Reading symbols from /usr/lib64/httpd/modules/mod_dav.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_dav.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_dav.so
Reading symbols from /usr/lib64/httpd/modules/mod_status.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_status.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_status.so
Reading symbols from /usr/lib64/httpd/modules/mod_autoindex.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_autoindex.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_autoindex.so
Reading symbols from /usr/lib64/httpd/modules/mod_info.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_info.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_info.so
Reading symbols from /usr/lib64/httpd/modules/mod_dav_fs.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_dav_fs.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_dav_fs.so
Reading symbols from /usr/lib64/httpd/modules/mod_vhost_alias.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_vhost_alias.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_vhost_alias.so
Reading symbols from /usr/lib64/httpd/modules/mod_negotiation.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_negotiation.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_negotiation.so
Reading symbols from /usr/lib64/httpd/modules/mod_dir.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_dir.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_dir.so
Reading symbols from /usr/lib64/httpd/modules/mod_actions.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_actions.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_actions.so
Reading symbols from /usr/lib64/httpd/modules/mod_speling.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_speling.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_speling.so
Reading symbols from /usr/lib64/httpd/modules/mod_userdir.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_userdir.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_userdir.so
Reading symbols from /usr/lib64/httpd/modules/mod_alias.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_alias.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_alias.so
Reading symbols from /usr/lib64/httpd/modules/mod_rewrite.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_rewrite.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_rewrite.so
Reading symbols from /usr/lib64/httpd/modules/mod_proxy.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_proxy.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_proxy.so
Reading symbols from /usr/lib64/httpd/modules/mod_proxy_balancer.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_proxy_balancer.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_proxy_balancer.so
Reading symbols from /usr/lib64/httpd/modules/mod_proxy_ftp.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_proxy_ftp.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_proxy_ftp.so
Reading symbols from /usr/lib64/httpd/modules/mod_proxy_http.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_proxy_http.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_proxy_http.so
Reading symbols from /usr/lib64/httpd/modules/mod_proxy_connect.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_proxy_connect.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_proxy_connect.so
Reading symbols from /usr/lib64/httpd/modules/mod_cache.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_cache.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_cache.so
Reading symbols from /usr/lib64/httpd/modules/mod_suexec.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_suexec.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_suexec.so
Reading symbols from /usr/lib64/httpd/modules/mod_disk_cache.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_disk_cache.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_disk_cache.so
Reading symbols from /usr/lib64/httpd/modules/mod_file_cache.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_file_cache.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_file_cache.so
Reading symbols from /usr/lib64/httpd/modules/mod_cgi.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_cgi.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_cgi.so
Reading symbols from /usr/lib64/httpd/modules/mod_version.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_version.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_version.so
Reading symbols from /usr/lib64/httpd/modules/mod_jk.so...done.
Loaded symbols for /usr/lib64/httpd/modules/mod_jk.so
Reading symbols from /usr/lib64/httpd/modules/mod_security2.so...done.
Loaded symbols for /etc/httpd/modules/mod_security2.so
Reading symbols from /usr/lib64/libxml2.so.2...done.
Loaded symbols for /usr/lib64/libxml2.so.2
Reading symbols from /usr/lib64/httpd/modules/mod_unique_id.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_unique_id.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_unique_id.so
Reading symbols from /usr/lib64/httpd/modules/mod_bw.so...done.
Loaded symbols for /usr/lib64/httpd/modules/mod_bw.so
Reading symbols from /usr/lib64/httpd/modules/mod_fcgid.so...done.
Loaded symbols for /usr/lib64/httpd/modules/mod_fcgid.so
Reading symbols from /usr/lib64/httpd/modules/mod_evasive20.so...done.
Loaded symbols for /etc/httpd/modules/mod_evasive20.so
Reading symbols from /usr/lib64/httpd/modules/mod_perl.so...done.
Loaded symbols for /etc/httpd/modules/mod_perl.so
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
Reading symbols from /lib64/libutil.so.1...done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /usr/lib64/httpd/modules/libphp5.so...done.
Loaded symbols for /etc/httpd/modules/libphp5.so
Reading symbols from /usr/lib64/libgmp.so.3...done.
Loaded symbols for /usr/lib64/libgmp.so.3
Reading symbols from /usr/lib64/libcurl.so.3...done.
Loaded symbols for /usr/lib64/libcurl.so.3
Reading symbols from /usr/lib64/libbz2.so.1...done.
Loaded symbols for /usr/lib64/libbz2.so.1
Reading symbols from /usr/lib64/libidn.so.11...done.
Loaded symbols for /usr/lib64/libidn.so.11
Reading symbols from /usr/lib64/httpd/modules/mod_proxy_ajp.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_proxy_ajp.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_proxy_ajp.so
Reading symbols from /usr/lib64/httpd/modules/mod_python.so...done.
Loaded symbols for /etc/httpd/modules/mod_python.so
Reading symbols from /usr/lib64/libpython2.4.so.1.0...done.
Loaded symbols for /usr/lib64/libpython2.4.so.1.0
Reading symbols from /usr/lib64/httpd/modules/mod_ssl.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_ssl.so.debug...done.
done.
Loaded symbols for /etc/httpd/modules/mod_ssl.so
Reading symbols from /usr/lib64/libdistcache.so.1...done.
Loaded symbols for /usr/lib64/libdistcache.so.1
Reading symbols from /usr/lib64/libnal.so.1...done.
Loaded symbols for /usr/lib64/libnal.so.1
Reading symbols from /usr/lib64/php/modules/apc.so...done.
Loaded symbols for /usr/lib64/php/modules/apc.so
Reading symbols from /usr/lib64/php/modules/bcmath.so...done.
Loaded symbols for /usr/lib64/php/modules/bcmath.so
Reading symbols from /usr/lib64/php/modules/dba.so...done.
Loaded symbols for /usr/lib64/php/modules/dba.so
Reading symbols from /usr/lib64/php/modules/dbase.so...done.
Loaded symbols for /usr/lib64/php/modules/dbase.so
Reading symbols from /usr/lib64/php/modules/dom.so...done.
Loaded symbols for /usr/lib64/php/modules/dom.so
Reading symbols from /usr/lib64/php/modules/gd.so...done.
Loaded symbols for /usr/lib64/php/modules/gd.so
Reading symbols from /usr/lib64/libt1.so.5...done.
Loaded symbols for /usr/lib64/libt1.so.5
Reading symbols from /usr/lib64/libfreetype.so.6...done.
Loaded symbols for /usr/lib64/libfreetype.so.6
Reading symbols from /usr/lib64/libX11.so.6...done.
Loaded symbols for /usr/lib64/libX11.so.6
Reading symbols from /usr/lib64/libXpm.so.4...done.
Loaded symbols for /usr/lib64/libXpm.so.4
Reading symbols from /usr/lib64/libpng12.so.0...done.
Loaded symbols for /usr/lib64/libpng12.so.0
Reading symbols from /usr/lib64/libjpeg.so.62...done.
Loaded symbols for /usr/lib64/libjpeg.so.62
Reading symbols from /usr/lib64/libXau.so.6...done.
Loaded symbols for /usr/lib64/libXau.so.6
Reading symbols from /usr/lib64/libXdmcp.so.6...done.
Loaded symbols for /usr/lib64/libXdmcp.so.6
Reading symbols from /usr/lib64/php/modules/imap.so...done.
Loaded symbols for /usr/lib64/php/modules/imap.so
Reading symbols from /usr/lib64/libc-client.so.1...done.
Loaded symbols for /usr/lib64/libc-client.so.1
Reading symbols from /lib64/libpam.so.0...done.
Loaded symbols for /lib64/libpam.so.0
Reading symbols from /lib64/libaudit.so.0...done.
Loaded symbols for /lib64/libaudit.so.0
Reading symbols from /usr/lib64/php/modules/json.so...done.
Loaded symbols for /usr/lib64/php/modules/json.so
Reading symbols from /usr/lib64/php/modules/ldap.so...done.
Loaded symbols for /usr/lib64/php/modules/ldap.so
Reading symbols from /usr/lib64/php/modules/mbstring.so...done.
Loaded symbols for /usr/lib64/php/modules/mbstring.so
Reading symbols from /usr/lib64/php/modules/mcrypt.so...done.
Loaded symbols for /usr/lib64/php/modules/mcrypt.so
Reading symbols from /usr/lib64/libmcrypt.so.4...done.
Loaded symbols for /usr/lib64/libmcrypt.so.4
Reading symbols from /usr/lib64/php/modules/mhash.so...done.
Loaded symbols for /usr/lib64/php/modules/mhash.so
Reading symbols from /usr/lib64/libmhash.so.2...done.
Loaded symbols for /usr/lib64/libmhash.so.2
Reading symbols from /usr/lib64/php/modules/mysql.so...done.
Loaded symbols for /usr/lib64/php/modules/mysql.so
Reading symbols from /usr/lib64/mysql/libmysqlclient.so.15...done.
Loaded symbols for /usr/lib64/mysql/libmysqlclient.so.15
Reading symbols from /usr/lib64/php/modules/mysqli.so...done.
Loaded symbols for /usr/lib64/php/modules/mysqli.so
Reading symbols from /usr/lib64/php/modules/ncurses.so...done.
Loaded symbols for /usr/lib64/php/modules/ncurses.so
Reading symbols from /usr/lib64/libpanel.so.5...done.
Loaded symbols for /usr/lib64/libpanel.so.5
Reading symbols from /usr/lib64/libncurses.so.5...done.
Loaded symbols for /usr/lib64/libncurses.so.5
Reading symbols from /usr/lib64/php/modules/odbc.so...done.
Loaded symbols for /usr/lib64/php/modules/odbc.so
Reading symbols from /usr/lib64/libodbc.so.1...done.
Loaded symbols for /usr/lib64/libodbc.so.1
Reading symbols from /usr/lib64/libodbcpsql.so.2...done.
Loaded symbols for /usr/lib64/libodbcpsql.so.2
Reading symbols from /usr/lib64/php/modules/pdo.so...done.
Loaded symbols for /usr/lib64/php/modules/pdo.so
Reading symbols from /usr/lib64/php/modules/pdo_mysql.so...done.
Loaded symbols for /usr/lib64/php/modules/pdo_mysql.so
Reading symbols from /usr/lib64/php/modules/pdo_odbc.so...done.
Loaded symbols for /usr/lib64/php/modules/pdo_odbc.so
Reading symbols from /usr/lib64/php/modules/pdo_pgsql.so...done.
Loaded symbols for /usr/lib64/php/modules/pdo_pgsql.so
Reading symbols from /usr/lib64/php/modules/pdo_sqlite.so...done.
Loaded symbols for /usr/lib64/php/modules/pdo_sqlite.so
Reading symbols from /usr/lib64/php/modules/pgsql.so...done.
Loaded symbols for /usr/lib64/php/modules/pgsql.so
Reading symbols from /usr/lib64/php/modules/pspell.so...done.
Loaded symbols for /usr/lib64/php/modules/pspell.so
Reading symbols from /usr/lib64/libaspell.so.15...done.
Loaded symbols for /usr/lib64/libaspell.so.15
Reading symbols from /usr/lib64/libpspell.so.15...done.
Loaded symbols for /usr/lib64/libpspell.so.15
Reading symbols from /usr/lib64/libstdc++.so.6...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /usr/lib64/php/modules/snmp.so...done.
Loaded symbols for /usr/lib64/php/modules/snmp.so
Reading symbols from /usr/lib64/libnetsnmp.so.10...done.
Loaded symbols for /usr/lib64/libnetsnmp.so.10
Reading symbols from /lib64/libwrap.so.0...done.
Loaded symbols for /lib64/libwrap.so.0
Reading symbols from /usr/lib64/php/modules/soap.so...done.
Loaded symbols for /usr/lib64/php/modules/soap.so
Reading symbols from /usr/lib64/php/modules/sqlite.so...done.
Loaded symbols for /usr/lib64/php/modules/sqlite.so
Reading symbols from /usr/lib64/php/modules/suhosin.so...done.
Loaded symbols for /usr/lib64/php/modules/suhosin.so
Reading symbols from /usr/lib64/php/modules/tidy.so...done.
Loaded symbols for /usr/lib64/php/modules/tidy.so
Reading symbols from /usr/lib64/libtidy-0.99.so.0...done.
Loaded symbols for /usr/lib64/libtidy-0.99.so.0
Reading symbols from /usr/lib64/php/modules/xdebug.so...done.
Loaded symbols for /usr/lib64/php/modules/xdebug.so
Reading symbols from /usr/lib64/php/modules/xmlreader.so...done.
Loaded symbols for /usr/lib64/php/modules/xmlreader.so
Reading symbols from /usr/lib64/php/modules/xmlrpc.so...done.
Loaded symbols for /usr/lib64/php/modules/xmlrpc.so
Reading symbols from /usr/lib64/php/modules/xmlwriter.so...done.
Loaded symbols for /usr/lib64/php/modules/xmlwriter.so
Reading symbols from /usr/lib64/php/modules/xsl.so...done.
Loaded symbols for /usr/lib64/php/modules/xsl.so
Reading symbols from /usr/lib64/libexslt.so.0...done.
Loaded symbols for /usr/lib64/libexslt.so.0
Reading symbols from /usr/lib64/libxslt.so.1...done.
Loaded symbols for /usr/lib64/libxslt.so.1
Reading symbols from /usr/lib64/libgcrypt.so.11...done.
Loaded symbols for /usr/lib64/libgcrypt.so.11
Reading symbols from /usr/lib64/libgpg-error.so.0...done.
Loaded symbols for /usr/lib64/libgpg-error.so.0
Reading symbols from /usr/lib64/php/modules/zip.so...done.
Loaded symbols for /usr/lib64/php/modules/zip.so
Reading symbols from /lib64/libnss_dns.so.2...done.
Loaded symbols for /lib64/libnss_dns.so.2
Reading symbols from /usr/lib64/gconv/ISO8859-1.so...done.
Loaded symbols for /usr/lib64/gconv/ISO8859-1.so
Core was generated by `/usr/sbin/httpd'.
Program terminated with signal 11, Segmentation fault.
[New process 11507]
#0 0x000069c3b6254006 in do_bind_function () from /etc/httpd/modules/libphp5.so
I have lots of core dumps, many of which give the same output.
Other programs causing sementation fault are:
Core was generated by `/usr/libexec/paxtest/anonmap'.
Core was generated by `/usr/libexec/paxtest/execbss'.
Core was generated by `/usr/libexec/paxtest/execdata'.
Core was generated by `/usr/libexec/paxtest/execheap'.
Core was generated by `/usr/libexec/paxtest/execstack'.
Core was generated by `/usr/libexec/paxtest/mprotanon'.
Core was generated by `/usr/libexec/paxtest/mprotbss'.
Core was generated by `/usr/libexec/paxtest/mprotdata'.
Core was generated by `/usr/libexec/paxtest/mprotheap'.
Core was generated by `/usr/libexec/paxtest/mprotshbss'.
Core was generated by `/usr/libexec/paxtest/mprotshdata'.
Core was generated by `/usr/libexec/paxtest/mprotstack'.
Core was generated by `/usr/libexec/paxtest/shlibbss'.
Core was generated by `/usr/libexec/paxtest/shlibdata'.
Re: Apache segmentation fault
Posted: Tue Mar 10, 2009 4:40 pm
by scott
Yup, this has nothing to do with mod_security or ASL:
Program terminated with signal 11, Segmentation fault.
[New process 11507]
#0 0x000069c3b6254006 in do_bind_function () from /etc/httpd/modules/libphp5.so
PHP died for some reason
Re: Apache segmentation fault
Posted: Tue Mar 10, 2009 5:41 pm
by faris
Just to keep people updated....
I got a couple of segfaults after a rule update a few days ago. No rule processing failures.
aus-city --- have you tried disabling the domain-blacklist.txt checks? That 100% sorted the problem for me previously. I'm currently running with them enabled, however. The thing is...they never seem to get fired. So they are much less important than anything else *for our sites*.
Faris.