Page 1 of 1

Strange httpd activity.

Posted: Wed Aug 18, 2010 5:09 am
by bananapar
A few times now our server load has suddenly started to climb dramatically, as has the use of swap, bringing it to its knees and making it appear to go down to website and webmail users.

We have managed to stop this, each time by doing an lsof -i :80 (after waiting for some time for ssh to respond) and seeing that httpd is creating abnormal amounts of processes to listen in on ports.
We then have to kill all httpd processes and restart httpd.

Unfortunately we have been unable to work out what seems to be triggering this. I have looked at various error and access logs which have shown no unusual activity around the time that the issues started and I am running out of ideas. This problem has only recently emerged and may be something to do with recent updates, as we didn't really experience this until after the ASL 2.2.10 upgrade.

Any ideas you might have would be most welcome.

Re: Strange httpd activity.

Posted: Wed Aug 18, 2010 5:33 am
by breun
Maybe Apache's MaxClients setting is too much for your machine to handle?

http://httpd.apache.org/docs/2.2/mod/mp ... maxclients

Google for tuning Apache MaxClients and you should find some info.

Re: Strange httpd activity.

Posted: Wed Aug 18, 2010 6:01 am
by bananapar
hi Breun,
Thank you for the reply.

I have considered this but that does not appear to be the cause.
I have also increased maxclients a few times before and think it may be unwise to increase it further at this point.

Re: Strange httpd activity.

Posted: Wed Aug 18, 2010 6:29 am
by breun
My suggestion actually was that it may have been set too *high* for what your hardware can handle.

Re: Strange httpd activity.

Posted: Wed Aug 18, 2010 12:41 pm
by bananapar
Oh yes sorry breun I must have misread your post.
Sadly this still doesn't appear to be the cause

Re: Strange httpd activity.

Posted: Tue Aug 24, 2010 7:59 am
by bananapar
This still occurs, though less frequently than when I first posted.

It does only happen during working hours so may be less to do with general traffic and more to do with something that either we or one of our clients is doing related to the server.
Still no luck so far with finding anything in common between the times it happened.

Re: Strange httpd activity.

Posted: Tue Aug 24, 2010 9:25 am
by scott
mod_rewrite rules possibly? Look around for .htaccess files

Re: Strange httpd activity.

Posted: Wed Aug 25, 2010 9:46 am
by bananapar
I've had a look and can't see anything abnormal with regards to the .htaccess files relating to mod_rewrite.
However, I don't know a huge amount about mod_rewrite and htaccess (what I do know is gleaned from googling it) so I could be missing something.

Re: Strange httpd activity.

Posted: Wed Aug 25, 2010 3:25 pm
by mneese77
most of your dynamic sites such as joomla, wordpress, forums, etc will have an .htaccess file in the site's root directory, and in most cases this has directives for mod-rewrite....

Re: Strange httpd activity.

Posted: Wed Dec 28, 2011 1:45 pm
by IMspintheweb
scott wrote:mod_rewrite rules possibly? Look around for .htaccess files
I am thinking I am running into high loads due to apache spawning way too many children, that being said - what's the deal with rewrite rules in .htaccess?

I have quite a few sites that use mod_rewrite for sef - does this conflict with anything in asl?

I turned off mod_security (I actually turned off WAF) and so far the server load has not skyrocketed, though it is a tad high @ 1.45 but thats MUCH better than the 40 i got woke up to this morning

:)

Re: Strange httpd activity.

Posted: Wed Dec 28, 2011 4:46 pm
by mikeshinn
I am thinking I am running into high loads due to apache spawning way too many children, that being said - what's the deal with rewrite rules in .htaccess?
mod_rewrite rules don't have anything to do with ASL (so there is no conflict). Poorly written rewrite rules can kill your system, they can cause loops, wasted cycles, etc. rewrite rules can be VERY cpu intensive.

If you disabled modsecurity and that helped your load, that sounds like you may:

1) be running low on memory
2) if load without mod_security is already at 1+, then your CPU is overworked already. Whats the hardware on your system? It sounds like you may already be saturating your system.
3) If the system is virtualized, you actual CPU utilization may be limited (and in such a way that you cant see it) - this can have horrifying effects on performance
4) if you have modsec enabled before you installed ASL, you may have a very inefficient configuration - were you running modsec before you installed ASL?
5) Did you enable any RBL or URI modsec rules? These require a fast local DNS server, if you don't have that your load will suffer accordingly
6) recompiled modsecurity module - some products, like cpanel, may replace the optimized modsec module ASL installs with a non-optimized module. Are you using cpanel or a third parties modsec module?
7) are you using a compiled apache, or an optimized apache provided by your OS vendor? If the former, us your OS vendors apache build. Source builds are hard to get optimized correctly.

Re: Strange httpd activity.

Posted: Wed Dec 28, 2011 5:45 pm
by IMspintheweb
copied this to https://www.atomicorp.com/forums/viewto ... 530#p32530 so it'd be a single thread...

:)



mikeshinn wrote:
I am thinking I am running into high loads due to apache spawning way too many children, that being said - what's the deal with rewrite rules in .htaccess?
mod_rewrite rules don't have anything to do with ASL (so there is no conflict). Poorly written rewrite rules can kill your system, they can cause loops, wasted cycles, etc. rewrite rules can be VERY cpu intensive.

If you disabled modsecurity and that helped your load, that sounds like you may:

1) be running low on memory -
2) if load without mod_security is already at 1+, then your CPU is overworked already. Whats the hardware on your system? It sounds like you may already be saturating your system.
3) If the system is virtualized, you actual CPU utilization may be limited (and in such a way that you cant see it) - this can have horrifying effects on performance
4) if you have modsec enabled before you installed ASL, you may have a very inefficient configuration - were you running modsec before you installed ASL?
5) Did you enable any RBL or URI modsec rules? These require a fast local DNS server, if you don't have that your load will suffer accordingly
6) recompiled modsecurity module - some products, like cpanel, may replace the optimized modsec module ASL installs with a non-optimized module. Are you using cpanel or a third parties modsec module?
7) are you using a compiled apache, or an optimized apache provided by your OS vendor? If the former, us your OS vendors apache build. Source builds are hard to get optimized correctly.

1 - definitely running low on memory

2 - runs a bit above 1 w/out mod_sec - you are right about that - system is over loaded...

3 - not virtualized - is a root server @ 1and1 here the specs:

dedicated single core root server @ 1and1 w/CentOS 5 with 1 gig ram, Parallels Plesk Panel 10.3.1 (64-bit)

CPU AuthenticAMD, AMD Athlon(tm) 64 Processor 3500+
Version Parallels Plesk Panel v10.3.1_build1013110726.09 os_CentOS 5

5 - dunno - will look at that...

6 - no mod_security before ASL

7 - it's using the apache that came with plesk 10.3.1: 2.2.3-53.el5.centos