Strange httpd activity.

General Discussion of atomic repo and development projects.

Ask for help here with anything else not covered by other forums.
bananapar
Forum User
Forum User
Posts: 38
Joined: Fri Jul 23, 2010 10:03 am
Location: UK

Strange httpd activity.

Unread post by bananapar »

A few times now our server load has suddenly started to climb dramatically, as has the use of swap, bringing it to its knees and making it appear to go down to website and webmail users.

We have managed to stop this, each time by doing an lsof -i :80 (after waiting for some time for ssh to respond) and seeing that httpd is creating abnormal amounts of processes to listen in on ports.
We then have to kill all httpd processes and restart httpd.

Unfortunately we have been unable to work out what seems to be triggering this. I have looked at various error and access logs which have shown no unusual activity around the time that the issues started and I am running out of ideas. This problem has only recently emerged and may be something to do with recent updates, as we didn't really experience this until after the ASL 2.2.10 upgrade.

Any ideas you might have would be most welcome.
breun
Long Time Forum Regular
Long Time Forum Regular
Posts: 2813
Joined: Sat Aug 20, 2005 9:30 am
Location: The Netherlands

Re: Strange httpd activity.

Unread post by breun »

Maybe Apache's MaxClients setting is too much for your machine to handle?

http://httpd.apache.org/docs/2.2/mod/mp ... maxclients

Google for tuning Apache MaxClients and you should find some info.
Lemonbit Internet Dedicated Server Management
bananapar
Forum User
Forum User
Posts: 38
Joined: Fri Jul 23, 2010 10:03 am
Location: UK

Re: Strange httpd activity.

Unread post by bananapar »

hi Breun,
Thank you for the reply.

I have considered this but that does not appear to be the cause.
I have also increased maxclients a few times before and think it may be unwise to increase it further at this point.
breun
Long Time Forum Regular
Long Time Forum Regular
Posts: 2813
Joined: Sat Aug 20, 2005 9:30 am
Location: The Netherlands

Re: Strange httpd activity.

Unread post by breun »

My suggestion actually was that it may have been set too *high* for what your hardware can handle.
Lemonbit Internet Dedicated Server Management
bananapar
Forum User
Forum User
Posts: 38
Joined: Fri Jul 23, 2010 10:03 am
Location: UK

Re: Strange httpd activity.

Unread post by bananapar »

Oh yes sorry breun I must have misread your post.
Sadly this still doesn't appear to be the cause
bananapar
Forum User
Forum User
Posts: 38
Joined: Fri Jul 23, 2010 10:03 am
Location: UK

Re: Strange httpd activity.

Unread post by bananapar »

This still occurs, though less frequently than when I first posted.

It does only happen during working hours so may be less to do with general traffic and more to do with something that either we or one of our clients is doing related to the server.
Still no luck so far with finding anything in common between the times it happened.
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8355
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Re: Strange httpd activity.

Unread post by scott »

mod_rewrite rules possibly? Look around for .htaccess files
bananapar
Forum User
Forum User
Posts: 38
Joined: Fri Jul 23, 2010 10:03 am
Location: UK

Re: Strange httpd activity.

Unread post by bananapar »

I've had a look and can't see anything abnormal with regards to the .htaccess files relating to mod_rewrite.
However, I don't know a huge amount about mod_rewrite and htaccess (what I do know is gleaned from googling it) so I could be missing something.
mneese77
Forum Regular
Forum Regular
Posts: 137
Joined: Wed May 12, 2010 5:40 pm

Re: Strange httpd activity.

Unread post by mneese77 »

most of your dynamic sites such as joomla, wordpress, forums, etc will have an .htaccess file in the site's root directory, and in most cases this has directives for mod-rewrite....
IMspintheweb
Forum User
Forum User
Posts: 30
Joined: Tue Dec 27, 2011 12:27 pm
Location: Golden, CO

Re: Strange httpd activity.

Unread post by IMspintheweb »

scott wrote:mod_rewrite rules possibly? Look around for .htaccess files
I am thinking I am running into high loads due to apache spawning way too many children, that being said - what's the deal with rewrite rules in .htaccess?

I have quite a few sites that use mod_rewrite for sef - does this conflict with anything in asl?

I turned off mod_security (I actually turned off WAF) and so far the server load has not skyrocketed, though it is a tad high @ 1.45 but thats MUCH better than the 40 i got woke up to this morning

:)
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: Strange httpd activity.

Unread post by mikeshinn »

I am thinking I am running into high loads due to apache spawning way too many children, that being said - what's the deal with rewrite rules in .htaccess?
mod_rewrite rules don't have anything to do with ASL (so there is no conflict). Poorly written rewrite rules can kill your system, they can cause loops, wasted cycles, etc. rewrite rules can be VERY cpu intensive.

If you disabled modsecurity and that helped your load, that sounds like you may:

1) be running low on memory
2) if load without mod_security is already at 1+, then your CPU is overworked already. Whats the hardware on your system? It sounds like you may already be saturating your system.
3) If the system is virtualized, you actual CPU utilization may be limited (and in such a way that you cant see it) - this can have horrifying effects on performance
4) if you have modsec enabled before you installed ASL, you may have a very inefficient configuration - were you running modsec before you installed ASL?
5) Did you enable any RBL or URI modsec rules? These require a fast local DNS server, if you don't have that your load will suffer accordingly
6) recompiled modsecurity module - some products, like cpanel, may replace the optimized modsec module ASL installs with a non-optimized module. Are you using cpanel or a third parties modsec module?
7) are you using a compiled apache, or an optimized apache provided by your OS vendor? If the former, us your OS vendors apache build. Source builds are hard to get optimized correctly.
IMspintheweb
Forum User
Forum User
Posts: 30
Joined: Tue Dec 27, 2011 12:27 pm
Location: Golden, CO

Re: Strange httpd activity.

Unread post by IMspintheweb »

copied this to https://www.atomicorp.com/forums/viewto ... 530#p32530 so it'd be a single thread...

:)



mikeshinn wrote:
I am thinking I am running into high loads due to apache spawning way too many children, that being said - what's the deal with rewrite rules in .htaccess?
mod_rewrite rules don't have anything to do with ASL (so there is no conflict). Poorly written rewrite rules can kill your system, they can cause loops, wasted cycles, etc. rewrite rules can be VERY cpu intensive.

If you disabled modsecurity and that helped your load, that sounds like you may:

1) be running low on memory -
2) if load without mod_security is already at 1+, then your CPU is overworked already. Whats the hardware on your system? It sounds like you may already be saturating your system.
3) If the system is virtualized, you actual CPU utilization may be limited (and in such a way that you cant see it) - this can have horrifying effects on performance
4) if you have modsec enabled before you installed ASL, you may have a very inefficient configuration - were you running modsec before you installed ASL?
5) Did you enable any RBL or URI modsec rules? These require a fast local DNS server, if you don't have that your load will suffer accordingly
6) recompiled modsecurity module - some products, like cpanel, may replace the optimized modsec module ASL installs with a non-optimized module. Are you using cpanel or a third parties modsec module?
7) are you using a compiled apache, or an optimized apache provided by your OS vendor? If the former, us your OS vendors apache build. Source builds are hard to get optimized correctly.

1 - definitely running low on memory

2 - runs a bit above 1 w/out mod_sec - you are right about that - system is over loaded...

3 - not virtualized - is a root server @ 1and1 here the specs:

dedicated single core root server @ 1and1 w/CentOS 5 with 1 gig ram, Parallels Plesk Panel 10.3.1 (64-bit)

CPU AuthenticAMD, AMD Athlon(tm) 64 Processor 3500+
Version Parallels Plesk Panel v10.3.1_build1013110726.09 os_CentOS 5

5 - dunno - will look at that...

6 - no mod_security before ASL

7 - it's using the apache that came with plesk 10.3.1: 2.2.3-53.el5.centos
Post Reply