high server load after moving to ASL

[imadsani]

Hey,

I just moved a high traffic website to a server running ASL and have noticed significantly higher load averages since the move. PHP seems to be causing this. Server load times have roughly doubled compared to the old one, plus this server is running SSD's

I have no conclusive evidence that ASL is causing this, neither am I blaming it, but I want to know whether this is to be expected with ASL? I have other servers running with ASL but this is the only high traffic website.

Specs:
PHP 5.4.26
CentOS 6.5
AMD Octacore
32GB RAM
2 x 250GB SSD's (Raid 1)

high server load after moving to ASL

[mikeshinn]

No, that shouldnt be happening. Whats contributing to your load, is it I/O, memory (lack thereof) and swapping, CPU bound issues? Can you provide some sar data on your system to help us to understand the condition of your system?

And do you use suphp?

[imadsani]

I'm afraid I'm new to SAR, could you please tell me which command to run to get you the output?

[DarkF@der]

http://www.thegeekstuff.com/2011/03/sar-examples/

[imadsani]

I got caught up with work, will post SAR results tomorrow, but for now here is a screenshot of newrelic



Are these many apache threads normal?

I've been experiencing almost double loads since I moved to this server. I upgraded PHP to 5.4.26 before moving here, plus I've moved the DB to another server running Percona but still high loads.

Edit: Could the fact that I'm running code encrypted by Ioncube have anything to do with higher loads?

[mikeshinn]

ASL doesnt control the amount of threads apache creates, thats all set but your apache configuration.

[faris]

Hmmm... if my maths is right, we're talking about 80Mb per apache process according to your stats. That sounds about right with ASL as there is a lot loaded via the mod_security rules (I typically see 60Mb to 100Mb+ per process).

If you want to reduce the memory footprint you can disable some of the mod_sec elements and see how it affects your memory usage, but other than some of the minor rulesets that would kind of be defeating the object of using ASL

You can reduce the number of processes by reducing the maxclients setting in httpd.conf but this can potentially result in some visitors waiting longer than they might otherwise do (when maxclients is reached often).

Basically it is a matter of balancing maxclients to keep memory (and total load) under control while at the same time trying to make sure that maxclients is not reached very often. Unfortunately, with a high traffic server that can be difficult and I'm guessing 200+ is what you actually need

It might be worth installing and using atop to help identify the problem if it is not as simple as too much RAM being used. It will help you see if there are disk/swap/memory issues. You can also use iotop to look more closely at disk IO hogs.