HELP! Server Load averaging around 2

[jpkelly]

I have a Plesk 10 CentOs (dv) 4.0 server with Media Temple.
After the cron jobs last the server load continues to hover around 2.0. Usually the load averages under 0.1.
top does not show anything obvious no high CPU users. There are around 100 processes running and that amount of processes does not seem to have been a problem in the past.
Media Temple staff say the server is operating within normal range (except for high processes count)
Any ideas how to identify where the high load is coming from/what is causing it?
This is causing connectivity issues.

[mikeshinn]

Top should tell how your cycles are being used in some contexts, do you see alot of io wait, user time, system time?

[jpkelly]

OK I feel like a noob.
Not sure how to tell io wait, user time, system time from top (or can you get those from top?)
Let alone what good or bad numbers would be.

[Highland]

First off, be sure you're running the atomic channel
http://www.atomicorp.com/wiki/index.php/Getting_Started

Next, install atop

yum install atop

atop should be able to tell you what is using your resources over time. top gives you the current snapshot only.

[jpkelly]

I installed atop but when I run it I get:

Code: Select all

activate process accounting: Operation not permitted
warning: no process exit detection!
Error opening terminal: ansi.

[jpkelly]

Although I have a lot of processes, I am doubtful (and so is Media Temple support) that this is the cause of the large load increase I experienced.

here is the output of ps axf: (current)

Code: Select all

  PID TTY      STAT   TIME COMMAND
    1 ?        Ss     0:01 init [3]      
30152 ?        S<s    0:00 /sbin/udevd -d
30714 ?        Ss     0:09 syslogd -m 0
31791 ?        Ss     0:00 dbus-daemon --system
31851 ?        Ss     0:00 /usr/sbin/sshd
30607 ?        Ss     0:00  \_ sshd: root@pts/0 
30635 pts/0    Ss     0:00      \_ -bash
 1844 pts/0    R+     0:00          \_ ps axf
31916 ?        S      0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysqld.pid --basedir=/usr --user=mysql
32074 ?        Sl     0:52  \_ /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
 1464 ?        S      0:00 /usr/lib/courier-imap/couriertcpd -address=0 -stderrlogger=/usr/sbin/courierlogger -stderrloggername=imapd -maxprocs=80 -maxperip=40 -pid=/var/run/imapd.pid -nodnslookup -noidentlookup 143 /usr/sbin/imapl
 4018 ?        S      0:00  \_ /usr/bin/imapd Maildir
 7582 ?        S      0:00  \_ /usr/bin/imapd Maildir
 7586 ?        S      0:00  \_ /usr/bin/imapd Maildir
 7687 ?        S      0:00  \_ /usr/bin/imapd Maildir
 7688 ?        S      0:00  \_ /usr/bin/imapd Maildir
 7738 ?        S      0:00  \_ /usr/bin/imapd Maildir
 7927 ?        S      0:00  \_ /usr/bin/imapd Maildir
31940 ?        S      0:00  \_ /usr/bin/imapd Maildir
16317 ?        S      0:00  \_ /usr/bin/imapd Maildir
16337 ?        S      0:00  \_ /usr/bin/imapd Maildir
17764 ?        S      0:00  \_ /usr/bin/imapd Maildir
29799 ?        S      0:00  \_ /usr/bin/imapd Maildir
 1466 ?        S      0:00 /usr/sbin/courierlogger imapd
 1478 ?        S      0:00 /usr/lib/courier-imap/couriertcpd -address=0 -stderrlogger=/usr/sbin/courierlogger -stderrloggername=imapd-ssl -maxprocs=80 -maxperip=40 -pid=/var/run/imapd-ssl.pid -nodnslookup -noidentlookup 993 /usr/bi
24323 ?        S      0:00  \_ /usr/bin/couriertls -server -tcpd /usr/sbin/imaplogin /usr/lib/courier-imap/authlib/authpsa /usr/bin/imapd Maildir
24352 ?        S      0:00  |   \_ /usr/bin/imapd Maildir
24478 ?        S      0:00  \_ /usr/bin/couriertls -server -tcpd /usr/sbin/imaplogin /usr/lib/courier-imap/authlib/authpsa /usr/bin/imapd Maildir
24495 ?        S      0:00  |   \_ /usr/bin/imapd Maildir
24522 ?        S      0:00  \_ /usr/bin/couriertls -server -tcpd /usr/sbin/imaplogin /usr/lib/courier-imap/authlib/authpsa /usr/bin/imapd Maildir
24533 ?        S      0:00  |   \_ /usr/bin/imapd Maildir
24561 ?        S      0:00  \_ /usr/bin/couriertls -server -tcpd /usr/sbin/imaplogin /usr/lib/courier-imap/authlib/authpsa /usr/bin/imapd Maildir
24570 ?        S      0:00  |   \_ /usr/bin/imapd Maildir
20367 ?        S      0:00  \_ /usr/bin/couriertls -server -tcpd /usr/sbin/imaplogin /usr/lib/courier-imap/authlib/authpsa /usr/bin/imapd Maildir
20384 ?        S      0:00  |   \_ /usr/bin/imapd Maildir
14188 ?        S      0:00  \_ /usr/bin/couriertls -server -tcpd /usr/sbin/imaplogin /usr/lib/courier-imap/authlib/authpsa /usr/bin/imapd Maildir
14234 ?        S      0:00  |   \_ /usr/bin/imapd Maildir
15386 ?        S      0:00  \_ /usr/bin/couriertls -server -tcpd /usr/sbin/imaplogin /usr/lib/courier-imap/authlib/authpsa /usr/bin/imapd Maildir
15460 ?        S      0:00      \_ /usr/bin/imapd Maildir
 1480 ?        S      0:00 /usr/sbin/courierlogger imapd-ssl
 1503 ?        S      0:00 /usr/lib/courier-imap/couriertcpd -address=0 -stderrlogger=/usr/sbin/courierlogger -stderrloggername=pop3d -maxprocs=40 -maxperip=4 -pid=/var/run/pop3d.pid -nodnslookup -noidentlookup 110 /usr/sbin/pop3lo
32719 ?        S      0:00  \_ /usr/bin/pop3d Maildir
 1505 ?        S      0:00 /usr/sbin/courierlogger pop3d
 1513 ?        S      0:00 /usr/lib/courier-imap/couriertcpd -address=0 -stderrlogger=/usr/sbin/courierlogger -stderrloggername=pop3d-ssl -maxprocs=40 -maxperip=4 -pid=/var/run/pop3d-ssl.pid -nodnslookup -noidentlookup 995 /usr/bin
30663 ?        S      0:00  \_ /usr/bin/couriertls -server -tcpd /usr/sbin/pop3login /usr/lib/courier-imap/authlib/authpsa /usr/bin/pop3d Maildir
30678 ?        S      0:00  |   \_ /usr/bin/pop3d Maildir
30672 ?        S      0:00  \_ /usr/bin/couriertls -server -tcpd /usr/sbin/pop3login /usr/lib/courier-imap/authlib/authpsa /usr/bin/pop3d Maildir
30691 ?        S      0:00      \_ /usr/bin/pop3d Maildir
 1515 ?        S      0:00 /usr/sbin/courierlogger pop3d-ssl
 1688 ?        Ss     0:04 /usr/bin/spamd --username=popuser --daemonize --nouser-config --helper-home-dir=/var/qmail --max-children 5 --create-prefs --virtual-config-dir=/var/qmail/mailnames/%d/%l/.spamassassin --pidfile=/var/run/
 1582 ?        S      1:16  \_ spamd child
 1398 ?        S      0:22  \_ spamd child
 7420 ?        Ss     0:00 crond
 7434 ?        Ss     0:00 /usr/sbin/munin-node
 7746 ?        Ss     0:00 xfs -droppriv -daemon
 7759 ?        Ss     0:00 /usr/bin/python /usr/lib/mailman/bin/mailmanctl -s -q start
 7829 ?        S      0:03  \_ /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=ArchRunner:0:1 -s
 7830 ?        S      0:02  \_ /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=BounceRunner:0:1 -s
 7831 ?        S      0:02  \_ /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=CommandRunner:0:1 -s
 7833 ?        S      0:03  \_ /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=IncomingRunner:0:1 -s
 7835 ?        S      0:02  \_ /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=NewsRunner:0:1 -s
 7836 ?        S      0:01  \_ /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=OutgoingRunner:0:1 -s
 7837 ?        S      0:02  \_ /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=VirginRunner:0:1 -s
 7838 ?        S      0:00  \_ /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=RetryRunner:0:1 -s
16088 ?        Ss     0:01 /usr/bin/sw-engine -c /usr/local/psa/admin/conf/php.ini /usr/local/psa/admin/bin/modules/watchdog/wdcollect -c /usr/local/psa/etc/modules/watchdog/wdcollect.inc.php
16089 ?        Ssl    0:08 /usr/local/psa/admin/bin/modules/watchdog/monit -Ic /usr/local/psa/etc/modules/watchdog/monitrc
21643 ?        S      0:09 /usr/sbin/sw-cp-serverd -f /etc/sw-cp-server/config
19974 ?        Ss     0:26 xinetd -stayalive -pidfile /var/run/xinetd.pid
32327 ?        Ss     0:00  \_ /usr/bin/spamdyke -f /etc/spamdyke.conf /var/qmail/bin/qmail-smtpd /var/qmail/bin/smtp_auth /var/qmail/bin/true /var/qmail/bin/cmd5checkpw /var/qmail/bin/true
32570 ?        S      0:00      \_ /var/qmail/bin/qmail-smtpd /var/qmail/bin/smtp_auth /var/qmail/bin/true /var/qmail/bin/cmd5checkpw /var/qmail/bin/true
22095 ?        Ssl    0:20 /usr/sbin/named -u named -c /etc/named.conf -u named -t /var/named/run-root
32676 ?        S      0:01 qmail-send
32678 ?        S      0:00  \_ splogger qmail
32679 ?        S      0:00  \_ qmail-lspawn | /usr/bin/deliverquota ./Maildir
32680 ?        S      0:00  \_ qmail-rspawn
32681 ?        S      0:00  \_ qmail-clean
26406 ?        Ss     0:01 /usr/sbin/httpd
26409 ?        S      0:00  \_ /usr/sbin/httpd
30435 ?        S      0:00  |   \_ /usr/bin/php-cgi
32047 ?        S      0:03  |   \_ /usr/bin/php-cgi
15367 ?        S      0:00  |   \_ /usr/bin/php-cgi
28033 ?        S      0:00  |   \_ /usr/bin/php-cgi
16051 ?        S      0:02  |   \_ /usr/bin/php-cgi
32478 ?        S      0:01  |   \_ /usr/bin/php-cgi
 7554 ?        S      0:00  |   \_ /usr/bin/php-cgi
17474 ?        S      0:00  |   \_ /usr/bin/php-cgi
22179 ?        S      0:11  \_ /usr/sbin/httpd
22427 ?        S      0:06  \_ /usr/sbin/httpd
 7569 ?        S      0:00  \_ /usr/sbin/httpd
 7929 ?        S      0:00  \_ /usr/sbin/httpd
21800 ?        S      0:00  \_ /usr/sbin/httpd

Normally (except for during cron jobs) the load is under 0.5.
Interesting thing is that the load increase began during or right after the nightly cron run. (Not that it was necessarily related to the cron jobs)
Here is what the spike looked like:

[biggles]

I installed atop but when I run it I get:

Code: Select all

activate process accounting: Operation not permitted
warning: no process exit detection!
Error opening terminal: ansi.

run it as root

[jpkelly]

run it as root

I am logged in as root.
using su root gives the same result.

[biggles]

Strange. I got the same error the other day and got around it by sudo atop

[AntiochInteractive]

I had similar problems and stopped running PHP via FastCGI. I used ruid2 (from Atomicorp) to get the functionality I needed from the FastCGI settings.

Hope this helps / relates in some way. I read through your running processes and it hinted that this might be something worth trying for you.

FastCGI leaked memory all over the place.

[jpkelly]

I had similar problems and stopped running PHP via FastCGI. I used ruid2 (from Atomicorp) to get the functionality I needed from the FastCGI settings.

Hope this helps / relates in some way. I read through your running processes and it hinted that this might be something worth trying for you.

FastCGI leaked memory all over the place.

Can you give some details about how you implemented ruid2?

[AntiochInteractive]

I'm running mod_ruid2, I installed with:

yum install mod_ruid2

and in:
/etc/httpd/conf.d/ruid2.conf

I'm adding each site I want to run as the FTP user.

--

Now here's the thing. I'm on PLESK 10.2 now and I'm seeing individual sites


<Directory /var/www/vhosts/moffettcommercialpainting.com/httpdocs>
RMode stat
RUidGid apache apache
RGroups apache psaserv
</Directory>
<Directory /var/www/vhosts/omnichurch.org/httpdocs>
RMode stat
RUidGid apache apache
RGroups apache psaserv
</Directory>
<Directory /var/www/vhosts/ahhmin.org/httpdocs>
RMode stat
RUidGid apache apache
RGroups apache psaserv
</Directory>


etc.

It seems to be a better solution.