mod_security memory usage

[scott]

No considerations, other than I have no faith in any virtualization technology to actually compartmentalize one virtual machine from another....

[Imaging]

Thanks. Dedicated it is.

[premierhosting]

Howdy,

Brought down loads successfully. Here's what I did:

1. Spamdyke.

75% of incoming is rejected before Spamassassin now. That has made a huge difference in loads.

2. rootcheck disable.

Rootcheck was spiking load every hour or so. I haven't figured out how to tell ASL to run rootcheck every say 24 hours, maybe at 3 a.m. when there are fewer website accesses.

Thanks.

[mikeshinn]

1. Spamdyke.

75% of incoming is rejected before Spamassassin now. That has made a huge difference in loads.

Spamassassin should not be run on an interactive web server. It can use an enormous amount of memory and CPU when scanning for spam - we do not recommend anyone run it on a webserver - instead, you should setup a dedicated box to filter out spam and never run spam filtering on an interactive web server. Setup firewall rules on your web server to only allow smtp in on ports 25 and 465 from your spam scrubbing boxes, and only allow users to use the submission port (587) with authentication. This will eliminate all spam filtering load problems on your web servers.

E-mail can stand to be delayed a few seconds (even a few minutes), so you should do all your email spam filtering on a box setaside for just that purpose. We created the free gamera project many years ago for just that reason and recommend you setup a spam scrubbing server, and run the gamera software there. Sooner or later you'll get a flood of mail in, and your box will suffer.

[premierhosting]

That's a good suggestion. In trying to find a cheap enough server to make it cost effective, what would you suggest for minimum ram requirements?

[mikeshinn]

Great question, so the good news is you dont need a very strong box to do the trick. Because email can be delayed a few seconds, and you are specializing the box to just srcub email, you can get away with a low end server. For example, we use basic P4s (just 2 threads) with just 1GB of RAM for our front end spam scrubbers and they dont even break a sweat.

[nobody]

I am very happy I see such a discussion. Even though we need at least 20 pages to cover this topic in my opinion.

My most loaded plesk box has 80 domains and around 1 million impressions a month. Handles all services plesk , mail , ftp etc. Has 4 GB of ram. Memory optimization is relatively very low. I use spamassasin. Haven't had a strong issue with the ram.
Also in that box I have a high load always due to mysql since the 1 site that carries the big load has a large database (few gigs) and executes to many queries on each impression. It takes at best 1.8 seconds to render the page with a dual 64bit proccessor on 2.6 ghz.

Basically in all plesk boxes I see ram flying away and swap being brutally used when plesk backups. Also clamd uses an enormous amount of ram. Other than that I think asl will run just fine with such a box.

Maybe something needs to be done with clamd I believe. The memory leaks of plesk will probably never be resolved...
Not to mention it will almost crash your box if you try to download the backup at your workstation. They say they have fixed that issue in 10 but I don't believe anymore what they announce.

[premierhosting]

Well, I'm still concerned about the memory usage here. Have my box at 69 load, then 89 load a couple minutes later.

Trying to find something other than bloated httpd to blame...
Some of the hanging out there processes are:

3120 ? Ss 0:00 crond
20425 ? S 0:00 \_ crond
20433 ? S 0:00 | \_ crond
20426 ? S 0:00 \_ crond
20432 ? Ss 0:00 | \_ /bin/sh -c /usr/local/psa/admin/bin/php /opt/plesk-billing/admin/sbin/runevents.php > /dev/null 2>&1
20719 ? R 0:00 | \_ /bin/sh -c /usr/local/psa/admin/bin/php /opt/plesk-billing/admin/sbin/runevents.php > /dev/null 2>&1
20644 ? S 0:00 \_ crond
20649 ? Ss 0:00 | \_ /bin/sh -c /var/ossec/active-response/bin/asl-shun.pl purge >/dev/null 2>&1
20717 ? S 0:00 | \_ /usr/bin/perl -w /var/ossec/active-response/bin/asl-shun.pl purge
20645 ? S 0:00 \_ crond
20650 ? Ds 0:00 | \_ /bin/sh -c /opt/drweb/update.pl
20646 ? S 0:00 \_ crond
20648 ? Ss 0:00 | \_ /bin/sh -c /usr/local/psa/admin/bin/php /opt/plesk-billing/admin/sbin/runevents.php > /dev/null 2>&1
20716 ? D 0:00 | \_ /usr/local/psa/admin/bin/php /opt/plesk-billing/admin/sbin/runevents.php

Any of those known to be resource killers?

The only service that seems to bring it back to sanity is to stop httpd. Looks like I have 70 httpd processes hanging out.

[nobody]

Well, I'm still concerned about the memory usage here. Have my box at 69 load, then 89 load a couple minutes later.

Trying to find something other than bloated httpd to blame...
Some of the hanging out there processes are:

3120 ? Ss 0:00 crond
20425 ? S 0:00 \_ crond
20433 ? S 0:00 | \_ crond
20426 ? S 0:00 \_ crond
20432 ? Ss 0:00 | \_ /bin/sh -c /usr/local/psa/admin/bin/php /opt/plesk-billing/admin/sbin/runevents.php > /dev/null 2>&1
20719 ? R 0:00 | \_ /bin/sh -c /usr/local/psa/admin/bin/php /opt/plesk-billing/admin/sbin/runevents.php > /dev/null 2>&1
20644 ? S 0:00 \_ crond
20649 ? Ss 0:00 | \_ /bin/sh -c /var/ossec/active-response/bin/asl-shun.pl purge >/dev/null 2>&1
20717 ? S 0:00 | \_ /usr/bin/perl -w /var/ossec/active-response/bin/asl-shun.pl purge
20645 ? S 0:00 \_ crond
20650 ? Ds 0:00 | \_ /bin/sh -c /opt/drweb/update.pl
20646 ? S 0:00 \_ crond
20648 ? Ss 0:00 | \_ /bin/sh -c /usr/local/psa/admin/bin/php /opt/plesk-billing/admin/sbin/runevents.php > /dev/null 2>&1
20716 ? D 0:00 | \_ /usr/local/psa/admin/bin/php /opt/plesk-billing/admin/sbin/runevents.php

Any of those known to be resource killers?

The only service that seems to bring it back to sanity is to stop httpd. Looks like I have 70 httpd processes hanging out.

run a ps -aux and paste it.

70 httpd processes ? Why on earth would you setup the httpd.conf with 70 max servers ??? This should be just fine in your httpd.conf

Code: Select all

<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  10000
</IfModule>

[premierhosting]

[root@cloud1 ~]# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 10348 720 ? Ss 12:38 0:00 init [3]
root 2 0.0 0.0 0 0 ? S< 12:38 0:00 [migration/0]
root 3 0.0 0.0 0 0 ? SN 12:38 0:00 [ksoftirqd/0]
root 4 0.0 0.0 0 0 ? S< 12:38 0:00 [watchdog/0]
root 5 0.0 0.0 0 0 ? S< 12:38 0:00 [events/0]
root 6 0.0 0.0 0 0 ? S< 12:38 0:00 [khelper]
root 23 0.0 0.0 0 0 ? S< 12:38 0:00 [kthread]
root 27 0.0 0.0 0 0 ? S< 12:38 0:10 [kblockd/0]
root 28 0.0 0.0 0 0 ? S< 12:38 0:00 [kacpid]
root 88 0.0 0.0 0 0 ? S< 12:38 0:00 [cqueue/0]
root 91 0.0 0.0 0 0 ? S< 12:38 0:00 [khubd]
root 93 0.0 0.0 0 0 ? S< 12:38 0:00 [kseriod]
root 159 0.1 0.0 0 0 ? S 12:38 0:46 [pdflush]
root 161 0.0 0.0 0 0 ? S< 12:38 0:03 [kswapd0]
root 162 0.0 0.0 0 0 ? S< 12:38 0:00 [aio/0]
root 308 0.0 0.0 0 0 ? S< 12:38 0:00 [kpsmoused]
root 325 0.0 0.0 0 0 ? S< 12:38 0:00 [ata/0]
root 326 0.0 0.0 0 0 ? S< 12:38 0:00 [ata_aux]
root 357 0.0 0.0 0 0 ? S< 12:38 0:00 [kstriped]
root 382 0.0 0.0 0 0 ? S< 12:38 0:00 [xfs_mru_cache]
root 383 0.0 0.0 0 0 ? S< 12:38 0:01 [xfslogd/0]
root 384 0.0 0.0 0 0 ? S< 12:38 0:00 [xfsdatad/0]
root 385 0.0 0.0 0 0 ? S< 12:39 0:06 [kjournald]
root 405 0.0 0.0 0 0 ? S< 12:39 0:00 [kauditd]
root 433 0.0 0.0 12604 356 ? S<s 12:39 0:00 /sbin/udevd -d
root 1101 0.0 0.0 0 0 ? S< 12:40 0:00 [kmpathd/0]
root 1102 0.0 0.0 0 0 ? S< 12:40 0:00 [kmpath_handlerd]
root 1127 0.0 0.0 0 0 ? S< 12:40 0:00 [xfsbufd]
root 1128 0.0 0.0 0 0 ? S< 12:40 0:00 [xfsaild]
root 1129 0.0 0.0 0 0 ? S< 12:40 0:00 [xfssyncd]
root 1131 0.0 0.0 0 0 ? S< 12:40 0:03 [xfsbufd]
root 1132 0.0 0.0 0 0 ? S< 12:40 0:00 [xfsaild]
root 1133 0.0 0.0 0 0 ? S< 12:55 0:02 [xfssyncd]
root 1510 0.0 0.0 6740 140 ? Ss 12:55 0:00 /sbin/dhclient -1 -q -lf /var/l
root 1544 0.0 0.0 5908 564 ? Ds 12:55 0:17 syslogd -m 0
root 1547 0.0 0.0 3804 432 ? Ss 12:55 0:12 klogd -x
root 1632 0.0 0.0 63516 868 ? Ss 12:56 0:00 /usr/sbin/sshd
ntp 1655 0.0 0.2 23388 5024 ? SLs 12:56 0:00 ntpd -u ntp:ntp -p /var/run/ntp
root 1667 0.0 0.1 100692 2884 ? Ss 12:56 0:00 sshd: russell [priv]
root 1706 0.0 0.0 65896 1152 ? S 12:56 0:00 /bin/sh /usr/bin/mysqld_safe --
mysql 1858 0.8 5.0 689428 103356 ? Sl 12:56 5:09 /usr/libexec/mysqld --basedir=/
russell 1888 0.0 0.0 100692 1260 ? S 12:56 0:01 sshd: russell@pts/0
russell 1895 0.0 0.0 66028 1236 pts/0 Ss 12:56 0:00 -bash
root 1994 0.0 0.0 132532 1948 pts/0 S 12:56 0:00 su -
root 2014 0.0 0.0 66132 1600 pts/0 S 12:56 0:00 -bash
root 2254 0.0 0.2 222524 5104 ? Ss 12:57 0:00 /var/asl/usr/sbin/asl-httpd
tortix 2299 0.0 0.5 228688 11272 ? S 12:57 0:00 /var/asl/usr/sbin/asl-httpd
named 2333 0.0 0.1 89024 2144 ? Ssl 12:57 0:00 /usr/sbin/named -u named -c /et
postgres 2373 0.0 0.0 120620 1744 ? S 12:57 0:00 /usr/bin/postmaster -p 5432 -D
postgres 2430 0.0 0.0 109800 276 ? S 12:58 0:00 postgres: logger process
postgres 2433 0.0 0.0 120620 400 ? S 12:58 0:00 postgres: writer process
postgres 2434 0.0 0.0 110800 264 ? S 12:58 0:00 postgres: stats buffer process
postgres 2435 0.0 0.0 109988 288 ? S 12:58 0:00 postgres: stats collector proce
root 9590 0.0 0.1 100692 3712 ? Ss 21:51 0:00 sshd: russell [priv]
russell 9612 0.0 0.0 100692 1960 ? S 21:52 0:00 sshd: russell@pts/2
russell 9613 0.0 0.0 66028 1584 pts/2 Ss 21:52 0:00 -bash
root 9710 0.0 0.1 132532 2588 pts/2 S 21:56 0:00 su -
root 9726 0.0 0.0 66028 1644 pts/2 S+ 21:57 0:00 -bash
root 11987 0.0 0.4 105812 8472 ? S 22:47 0:00 /usr/bin/perl -w /usr/bin/psmon
root 12008 0.0 0.1 100692 3700 ? Ss 22:50 0:00 sshd: russell [priv]
root 12043 0.0 0.1 33792 2292 ? S 22:52 0:00 /usr/bin/couriertls -server -tc
popuser 12044 0.0 0.0 10268 1036 ? S 22:52 0:00 /usr/bin/imapd Maildir
russell 12047 0.0 0.0 100692 1864 ? S 22:52 0:00 sshd: russell@pts/3
russell 12059 0.0 0.0 66028 1580 pts/3 Ss 22:52 0:00 -bash
root 12095 0.0 0.1 132532 2588 pts/3 S 22:52 0:00 su -
root 12114 2.9 0.0 66028 1676 pts/3 S 22:53 0:11 -bash
cacti 12169 0.0 0.1 133488 2764 ? S 22:55 0:00 crond
cacti 12170 0.0 0.0 8668 1064 ? Ss 22:55 0:00 /bin/sh -c /usr/bin/php /usr/sh
cacti 12172 0.0 0.8 163508 17128 ? S 22:55 0:00 /usr/bin/php /usr/share/cacti/p
root 12198 0.0 0.0 63932 1440 pts/0 S+ 22:56 0:00 /bin/sh /sbin/service httpd res
root 12203 0.0 0.0 66028 1580 pts/0 S+ 22:56 0:00 /bin/bash /etc/init.d/httpd res
root 12209 0.0 0.0 10760 1164 pts/0 S+ 22:56 0:00 /bin/bash -c ulimit -S -c 0 >/d
root 12210 14.4 0.4 225920 9580 pts/0 D+ 22:56 0:25 /usr/sbin/httpd
qmaild 12230 0.0 0.1 46784 3188 ? Ss 22:57 0:00 /usr/bin/spamdyke -f /etc/spamd
qmaild 12235 0.0 0.1 44564 2508 ? Ss 22:58 0:00 /usr/bin/spamdyke -f /etc/spamd
qmaild 12236 0.0 0.0 33672 1536 ? S 22:58 0:00 /var/qmail/bin/qmail-smtpd /var
qmaild 12247 0.0 0.0 15832 1200 ? S 22:59 0:00 bin/qmail-queue
root 12248 0.0 0.0 3676 364 ? Ss 22:59 0:00 /usr/sbin/rblsmtpd -r sbl.spamh
root 12258 0.0 0.4 179108 10100 ? Ss 22:59 0:00 /usr/bin/sw-engine-cgi -c /usr/
psaadm 12267 3.5 2.4 257048 49596 ? S 22:59 0:00 /usr/bin/sw-engine-cgi -c /usr/
psaadm 12268 0.0 0.1 179108 3884 ? S 22:59 0:00 /usr/bin/sw-engine-cgi -c /usr/
psaadm 12269 0.0 0.1 179108 3884 ? S 22:59 0:00 /usr/bin/sw-engine-cgi -c /usr/
qmaild 12270 0.0 0.0 33672 1496 ? S 22:59 0:00 /var/qmail/bin/qmail-smtpd /var
root 12274 0.0 0.0 65592 1000 pts/3 R+ 22:59 0:00 ps aux
root 13204 0.0 0.0 0 0 ? S< 13:06 0:00 [cifsoplockd]
root 13374 0.0 0.0 3788 440 tty1 Ss+ 13:07 0:00 /sbin/mingetty tty1
root 13375 0.0 0.0 3788 440 tty2 Ss+ 13:07 0:00 /sbin/mingetty tty2
root 13376 0.0 0.0 3788 440 tty3 Ss+ 13:07 0:00 /sbin/mingetty tty3
root 13377 0.0 0.0 3788 440 tty4 Ss+ 13:07 0:00 /sbin/mingetty tty4
root 13378 0.0 0.0 3788 440 tty5 Ss+ 13:07 0:00 /sbin/mingetty tty5
root 13379 0.0 0.0 3788 440 tty6 Ss+ 13:07 0:00 /sbin/mingetty tty6
root 13380 0.0 0.0 3800 488 ttyS0 Ss+ 13:07 0:00 /sbin/agetty -L ttyS0 57600 vt1
root 17376 0.0 0.0 0 0 ? S 13:59 0:00 [pdflush]
ossecm 20651 0.0 0.0 6060 248 ? S 15:00 0:00 /var/ossec/bin/ossec-maild
500 21475 0.0 0.1 60348 3488 ? S 15:43 0:07 /usr/sbin/sw-cp-serverd -f /etc
tortix 21889 0.0 0.3 222524 6716 ? S 15:56 0:00 /var/asl/usr/sbin/asl-httpd
tortix 21890 0.0 0.3 223548 8068 ? S 15:56 0:00 /var/asl/usr/sbin/asl-httpd
tortix 21892 0.0 0.3 223548 7504 ? S 15:56 0:00 /var/asl/usr/sbin/asl-httpd
root 21911 0.0 0.0 21644 912 ? Ss 15:57 0:01 xinetd -stayalive -pidfile /var
root 21942 0.0 0.0 74808 1184 ? Ss 15:57 0:00 crond
root 22047 0.0 0.1 52992 2292 ? Ssl 15:58 0:00 /usr/local/psa/admin/bin/module
tortix 22597 0.0 0.3 223548 7692 ? S 16:00 0:00 /var/asl/usr/sbin/asl-httpd
root 23328 0.0 0.0 13164 616 ? S 16:01 0:00 /usr/lib/courier-imap/couriertc
root 23330 0.0 0.0 3788 548 ? S 16:01 0:00 /usr/sbin/courierlogger imapd
root 23338 0.0 0.0 13164 612 ? S 16:01 0:00 /usr/lib/courier-imap/couriertc
root 23343 0.0 0.0 3788 552 ? S 16:01 0:00 /usr/sbin/courierlogger imapd-s
root 23346 0.0 0.0 13164 616 ? S 16:01 0:00 /usr/lib/courier-imap/couriertc
root 23348 0.0 0.0 3788 552 ? S 16:01 0:00 /usr/sbin/courierlogger pop3d
root 23355 0.0 0.0 13164 616 ? S 16:01 0:00 /usr/lib/courier-imap/couriertc
root 23359 0.0 0.0 3788 552 ? S 16:01 0:00 /usr/sbin/courierlogger pop3d-s
root 24498 0.0 2.4 171228 49928 ? Ss 16:14 0:01 /usr/bin/spamd -d -c -m5 -H -r
root 24502 0.0 2.3 171228 48000 ? S 16:14 0:00 spamd child
root 24503 0.0 2.3 171228 47912 ? S 16:14 0:00 spamd child
qmails 24522 0.0 0.0 3844 568 ? S 16:14 0:02 qmail-send
qmaill 24523 0.0 0.0 3796 548 ? S 16:14 0:00 splogger qmail
root 24524 0.0 0.0 3836 456 ? S 16:14 0:00 qmail-lspawn | /usr/bin/deliver
qmailr 24525 0.0 0.0 3836 464 ? S 16:14 0:00 qmail-rspawn
qmailq 24526 0.0 0.0 3792 412 ? S 16:14 0:00 qmail-clean
root 25825 0.1 9.7 275000 200608 ? Ssl 16:20 0:46 clamd

[premierhosting]

If anyone cares my problems are solved. Here are the steps:

1. Throw "Cloud Server" in the virtual garbage, cos that's what it is.
2. Bring up a real live dedicated server.
3. Install ASL.
4. Live happily ever after.

I did get load down by disabling most of the ASL features shortly before I killed the Cloud Server. It seemed mostly fine without the ASL stuff. My brand spankin' new actual server is handling everything effortlessly though, and I'm throwing more load at it. Ended up with 4gb ram instead of 2gb cloud ram, so that *may* be impacting, but from what I read 2gb of ram should have been OK.

Learn from my 6 months of hell. Don't install ASL on a Cloud if you value your free time.

[breun]

Learn from my 6 months of hell. Don't install ASL on a Cloud if you value your free time.

Too bad you had a bad experience, but ASL is working fine for us on virtual servers.

[premierhosting]

Mine was the Cloud style on Xen with 1and1. From what I understand, other virtualization platforms behave better. What are you on?

[breun]

We have clients on all kinds of virtualization platforms. Full virtualization technologies like VMware and KVM have the advantage that you can use the ASL kernel in the virtual servers, sadly with Xen's paravirtualization this is not possible. But either way, performance is fine if the virtual server has enough resources as its disposal (CPU, RAM, I/O), not any difference with the dedicated world as far as I know.

[scott]

Kind of wandering from the original topic, but we've actually been investigating a bug with Plesk 10 and Xen server in general. It manifests as very high IO usage sometimes leading to a full crash of the system.

The only definitive thing I can say about it at this point is that its related to the mail system. We did a fresh install, without ASL, with Plesk 10 w/ postfix and drweb. Those systems will consistently crash after 10 minutes at 3 different providers we tested it against. Turning postfix off stopped the problem in every case. Removing drweb also seems to make a huge impact, but its unclear if it is just exacerbating a different problem or is the root cause. We should probably start a different thread for this, but I'd love to get any other xen/plesk 10 integration information tracked so we can get that to citirix or parallels to investigate further.