Apache segmentation fault

[hostingguy]

So would you run this proactively such as to set this to run in a cron job every few minutes, or is this a reactive script to run manualy after the fact?

Is there any recomended usage instructions - any chance something like this could be ported into psmon?

[aus-city]

The script once running constantly monitors and restarts apache / httpd as soon as a rule process failed is detected.

You put the script into your rc.local so its run on bootup.

[diego]

I run the script

and got this:

[root@server1 ~]# ./restart_apache.pl
Possible unintended interpolation of @a in string at /root/restart_apache.pl line 60.
Unrecognized escape \s passed through at /root/restart_apache.pl line 60.
syntax error at /root/restart_apache.pl line 62, near "system"
Execution of /root/restart_apache.pl aborted due to compilation errors.

[aus-city]

Try the old script I am waiting for Michael to get back to me, this one works:

#!/usr/bin/perl -w
# Released under GPL v2
#
# If apache freaks out and modsec stops working kill apache and restart it.
#
use IO::File;
$|=1;

my $adminemail = "root\@localhost";

my $log = "/var/log/httpd/error_log";
my $alert = qr/ModSecurity.+Rule processing failed/i;

my $f;
my $lastsize = 100000000000;
while( 1 ) {
# if logfile rotates, reopen
my @s = stat $log;
my $size = $s[7];
if( $size < $lastsize ) {
# reopen
print "Reopening $log and standing at the end of the queue (s=$size, l=$lastsize).\n";
$f = new IO::File "< $log" or die $!;
$f->seek( 0, 2 );
}
$lastsize = $size;

# read new loglines and act on them
while( <$f> ) {
if( /$alert/ ) {
&act($_);
# seek to the end (skip lines between)
$f->seek( 0, 2 );

} else {
#print "skip '$_'\n";
}
}

# clear EOF (follow logfile)
$f->seek(0,1);
#print ".";
sleep 1;
}


sub act {
# restart apache
system "/etc/init.d/apache restart";
# email admin
system "echo 'Apache restart at " . scalar(localtime) . "' | mail $adminemail -s Apache-Restarted";
# don't restart again sooner than 5 minutes, in case of mad apache
sleep 300;
}

[aus-city]

Go back to the NEW script and use new sub act this until we get an update:

sub act {
# stop apache
system "/etc/init.d/httpd stop";
# clear semaphores for apache
# system "ipcs -s | grep $apacheuser | perl -e 'while (<STDIN>) { @a=split(/\s+/); print `ipcrm sem $a[1]`}'"
# clear all dead apache processes
system "killall httpd";
# start apache clean
system "/etc/init.d/httpd start";
# email admin
system "echo 'Service httpd restarted $HOSTNAME due to processing error at " . scalar(localtime) . "' | mail $adminemail -s Service httpd restarted from $HOSTNAME";

[mikeshinn]

New fixed script uploaded.

[aus-city]

Thanks Mike!

The other thing I would like to note, I just recently built httpd-2.2.11-1 from rawhide SRPM for FC8 and FC10

I must say that httpd-2.2.11 seems a heck of a lot more stable on this issue. Since running httpd-2.2.11 I have not seen one failure yet.

If you like I can send you the SRPM, as you got to play with it as now they moved the pid from /var/run/httpd.pid to /var/run/httpd/httpd.pid

Although this may not bother most systems, if you got plesk it does as watchdog is looking for /var/run/httpd.pid

I still let the rpm create the /var/run/httpd/ directory, but the pid is still in the old location. Naturally the init script and a few others had to be edited.

Yell and I can send it.

[aus-city]

httpd-2.2.11 seems to be going great guns! I have reenabled mod_mem_cache and no crashes yet.

I let it soak a few more days ,but 36 hours and going well!

[scott]

Do you think you can drop that box down to 1G of ram to test it? It would rule out the extra memory giving you more breathing room.

[aus-city]

Hi Scott,

Unfortunately no its a pair of 4GB sticks as the ram is interleaved

But its been almost 2 weeks now

Still stable.

[scott]

Did you try rolling back apr yet?

[faris]

I thought I'd update this thread with some interesting, but still confusing, info.

This only relates to Virtuozzo VPS users.

Parallels recently released an updated kernel and tools. After installing these, there has been a significant change in the behaviour of the VPSes I have running ASL.

Firstly there are no more glibc errors and no segfaults in the Apache error log. I can no longer produce a segfault on demand by rebooting a VPS and then accessing webmail.

Secondly, instead of slowly increasing over time, memory usage is relatively static. There are times when it will go above its norm, but it then goes down again and generally it is staying constant.

Thirdly, when apache is restarted (restart or graceful), memory is released. I do not think this was happening before. It always constantly went up and up and up. The amount released seems to depend on a lot of things -- I presume how many processes are running at the time.

*** I still have the domain blacklist ruleset disabled - I will shortly test with it enabled to see what happens ***

Faris.

[scott]

Really, thats good news. What kernel were you on before, and what are you on now? Ive been poking at the openvz code to include in future ASL kernels

[faris]

2.6.18-028stab060.8 currently.

2.6.18-028stab059.6 previously.

Unfortunately, although things are much better, when I enable the domain-blacklist.txt things do go wrong. glibc errors again. So far no segfaults but semaphores go up and stay up, memory use goes up and stays up.

When the semaphores start to go up, segfaults are sure to happen.

I've therefore disabled those rules again

Faris.

[scott]

Well thats a start, I can look at the changes between those two code bases and see if something relevant did change or not. As I mentioned before we are working on building openvz/virtuozzo capabilities into the ASL kernel. Its kind of an uphill battle though, since its kind of all mashed together with other patches, so we haven't gotten to the point of knowing if it can be done or not yet.