Strange httpd activity.
Strange httpd activity.
A few times now our server load has suddenly started to climb dramatically, as has the use of swap, bringing it to its knees and making it appear to go down to website and webmail users.
We have managed to stop this, each time by doing an lsof -i :80 (after waiting for some time for ssh to respond) and seeing that httpd is creating abnormal amounts of processes to listen in on ports.
We then have to kill all httpd processes and restart httpd.
Unfortunately we have been unable to work out what seems to be triggering this. I have looked at various error and access logs which have shown no unusual activity around the time that the issues started and I am running out of ideas. This problem has only recently emerged and may be something to do with recent updates, as we didn't really experience this until after the ASL 2.2.10 upgrade.
Any ideas you might have would be most welcome.
We have managed to stop this, each time by doing an lsof -i :80 (after waiting for some time for ssh to respond) and seeing that httpd is creating abnormal amounts of processes to listen in on ports.
We then have to kill all httpd processes and restart httpd.
Unfortunately we have been unable to work out what seems to be triggering this. I have looked at various error and access logs which have shown no unusual activity around the time that the issues started and I am running out of ideas. This problem has only recently emerged and may be something to do with recent updates, as we didn't really experience this until after the ASL 2.2.10 upgrade.
Any ideas you might have would be most welcome.
-
- Long Time Forum Regular
- Posts: 2813
- Joined: Sat Aug 20, 2005 9:30 am
- Location: The Netherlands
Re: Strange httpd activity.
Maybe Apache's MaxClients setting is too much for your machine to handle?
http://httpd.apache.org/docs/2.2/mod/mp ... maxclients
Google for tuning Apache MaxClients and you should find some info.
http://httpd.apache.org/docs/2.2/mod/mp ... maxclients
Google for tuning Apache MaxClients and you should find some info.
Lemonbit Internet Dedicated Server Management
Re: Strange httpd activity.
hi Breun,
Thank you for the reply.
I have considered this but that does not appear to be the cause.
I have also increased maxclients a few times before and think it may be unwise to increase it further at this point.
Thank you for the reply.
I have considered this but that does not appear to be the cause.
I have also increased maxclients a few times before and think it may be unwise to increase it further at this point.
-
- Long Time Forum Regular
- Posts: 2813
- Joined: Sat Aug 20, 2005 9:30 am
- Location: The Netherlands
Re: Strange httpd activity.
My suggestion actually was that it may have been set too *high* for what your hardware can handle.
Lemonbit Internet Dedicated Server Management
Re: Strange httpd activity.
Oh yes sorry breun I must have misread your post.
Sadly this still doesn't appear to be the cause
Sadly this still doesn't appear to be the cause
Re: Strange httpd activity.
This still occurs, though less frequently than when I first posted.
It does only happen during working hours so may be less to do with general traffic and more to do with something that either we or one of our clients is doing related to the server.
Still no luck so far with finding anything in common between the times it happened.
It does only happen during working hours so may be less to do with general traffic and more to do with something that either we or one of our clients is doing related to the server.
Still no luck so far with finding anything in common between the times it happened.
-
- Atomicorp Staff - Site Admin
- Posts: 8355
- Joined: Wed Dec 31, 1969 8:00 pm
- Location: earth
- Contact:
Re: Strange httpd activity.
mod_rewrite rules possibly? Look around for .htaccess files
Re: Strange httpd activity.
I've had a look and can't see anything abnormal with regards to the .htaccess files relating to mod_rewrite.
However, I don't know a huge amount about mod_rewrite and htaccess (what I do know is gleaned from googling it) so I could be missing something.
However, I don't know a huge amount about mod_rewrite and htaccess (what I do know is gleaned from googling it) so I could be missing something.
Re: Strange httpd activity.
most of your dynamic sites such as joomla, wordpress, forums, etc will have an .htaccess file in the site's root directory, and in most cases this has directives for mod-rewrite....
-
- Forum User
- Posts: 30
- Joined: Tue Dec 27, 2011 12:27 pm
- Location: Golden, CO
Re: Strange httpd activity.
I am thinking I am running into high loads due to apache spawning way too many children, that being said - what's the deal with rewrite rules in .htaccess?scott wrote:mod_rewrite rules possibly? Look around for .htaccess files
I have quite a few sites that use mod_rewrite for sef - does this conflict with anything in asl?
I turned off mod_security (I actually turned off WAF) and so far the server load has not skyrocketed, though it is a tad high @ 1.45 but thats MUCH better than the 40 i got woke up to this morning
- mikeshinn
- Atomicorp Staff - Site Admin
- Posts: 4149
- Joined: Thu Feb 07, 2008 7:49 pm
- Location: Chantilly, VA
Re: Strange httpd activity.
mod_rewrite rules don't have anything to do with ASL (so there is no conflict). Poorly written rewrite rules can kill your system, they can cause loops, wasted cycles, etc. rewrite rules can be VERY cpu intensive.I am thinking I am running into high loads due to apache spawning way too many children, that being said - what's the deal with rewrite rules in .htaccess?
If you disabled modsecurity and that helped your load, that sounds like you may:
1) be running low on memory
2) if load without mod_security is already at 1+, then your CPU is overworked already. Whats the hardware on your system? It sounds like you may already be saturating your system.
3) If the system is virtualized, you actual CPU utilization may be limited (and in such a way that you cant see it) - this can have horrifying effects on performance
4) if you have modsec enabled before you installed ASL, you may have a very inefficient configuration - were you running modsec before you installed ASL?
5) Did you enable any RBL or URI modsec rules? These require a fast local DNS server, if you don't have that your load will suffer accordingly
6) recompiled modsecurity module - some products, like cpanel, may replace the optimized modsec module ASL installs with a non-optimized module. Are you using cpanel or a third parties modsec module?
7) are you using a compiled apache, or an optimized apache provided by your OS vendor? If the former, us your OS vendors apache build. Source builds are hard to get optimized correctly.
Michael Shinn
Atomicorp - Security For Everyone
Atomicorp - Security For Everyone
-
- Forum User
- Posts: 30
- Joined: Tue Dec 27, 2011 12:27 pm
- Location: Golden, CO
Re: Strange httpd activity.
copied this to https://www.atomicorp.com/forums/viewto ... 530#p32530 so it'd be a single thread...
1 - definitely running low on memory
2 - runs a bit above 1 w/out mod_sec - you are right about that - system is over loaded...
3 - not virtualized - is a root server @ 1and1 here the specs:
dedicated single core root server @ 1and1 w/CentOS 5 with 1 gig ram, Parallels Plesk Panel 10.3.1 (64-bit)
CPU AuthenticAMD, AMD Athlon(tm) 64 Processor 3500+
Version Parallels Plesk Panel v10.3.1_build1013110726.09 os_CentOS 5
5 - dunno - will look at that...
6 - no mod_security before ASL
7 - it's using the apache that came with plesk 10.3.1: 2.2.3-53.el5.centos
mikeshinn wrote:mod_rewrite rules don't have anything to do with ASL (so there is no conflict). Poorly written rewrite rules can kill your system, they can cause loops, wasted cycles, etc. rewrite rules can be VERY cpu intensive.I am thinking I am running into high loads due to apache spawning way too many children, that being said - what's the deal with rewrite rules in .htaccess?
If you disabled modsecurity and that helped your load, that sounds like you may:
1) be running low on memory -
2) if load without mod_security is already at 1+, then your CPU is overworked already. Whats the hardware on your system? It sounds like you may already be saturating your system.
3) If the system is virtualized, you actual CPU utilization may be limited (and in such a way that you cant see it) - this can have horrifying effects on performance
4) if you have modsec enabled before you installed ASL, you may have a very inefficient configuration - were you running modsec before you installed ASL?
5) Did you enable any RBL or URI modsec rules? These require a fast local DNS server, if you don't have that your load will suffer accordingly
6) recompiled modsecurity module - some products, like cpanel, may replace the optimized modsec module ASL installs with a non-optimized module. Are you using cpanel or a third parties modsec module?
7) are you using a compiled apache, or an optimized apache provided by your OS vendor? If the former, us your OS vendors apache build. Source builds are hard to get optimized correctly.
1 - definitely running low on memory
2 - runs a bit above 1 w/out mod_sec - you are right about that - system is over loaded...
3 - not virtualized - is a root server @ 1and1 here the specs:
dedicated single core root server @ 1and1 w/CentOS 5 with 1 gig ram, Parallels Plesk Panel 10.3.1 (64-bit)
CPU AuthenticAMD, AMD Athlon(tm) 64 Processor 3500+
Version Parallels Plesk Panel v10.3.1_build1013110726.09 os_CentOS 5
5 - dunno - will look at that...
6 - no mod_security before ASL
7 - it's using the apache that came with plesk 10.3.1: 2.2.3-53.el5.centos