Qmail Under attack

tungsten · Unread post by **tungsten** » Wed May 28, 2008 9:43 pm

Hi everyone,

We've been under attack for the last several hours. Our top process looks like this:

op - 20:39:44 up  9:20,  2 users,  load average: 41.20, 38.51, 36.41
Tasks: 199 total,  42 running, 157 sleeping,   0 stopped,   0 zombie
Cpu(s): 99.6% us,  0.4% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:   4148848k total,  2298032k used,  1850816k free,   159212k buffers
Swap:  2096472k total,        0k used,  2096472k free,  1733328k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                      
 2182 qmaild    25   0  4264 1380 1164 R   23  0.0   1:28.99 qmail-smtpd                                                                                                                                   
 3824 qmaild    25   0  5272 1380 1164 R   21  0.0   0:04.14 qmail-smtpd                                                                                                                                   
 1954 mysql     15   0  662m 112m 4848 S   21  2.8  77:45.83 mysqld                                                                                                                                        
 2042 qmaild    25   0  5468 1376 1164 R   21  0.0   1:34.31 qmail-smtpd                                                                                                                                   
 3641 qmaild    25   0  3460 1372 1164 R   21  0.0   0:14.45 qmail-smtpd                                                                                                                                   
  947 qmaild    25   0  3808 1384 1164 R   20  0.0   2:30.10 qmail-smtpd                                                                                                                                   
 2476 qmaild    25   0  4416 1380 1164 R   20  0.0   1:19.89 qmail-smtpd                                                                                                                                   
 3689 qmaild    25   0  3660 1376 1164 R   20  0.0   0:12.28 qmail-smtpd                                                                                                                                   
25355 qmaild    25   0  5344 1380 1164 R   20  0.0  10:47.90 qmail-smtpd                                                                                                                                   
 1085 qmaild    25   0  4196 1376 1164 R   20  0.0   2:23.28 qmail-smtpd                                                                                                                                   
 1681 qmaild    25   0  5136 1388 1164 R   20  0.0   1:50.48 qmail-smtpd                                                                                                                                   
 2004 qmaild    25   0  4316 1376 1164 R   20  0.0   1:40.92 qmail-smtpd                                                                                                                                   
 2111 qmaild    25   0  5020 1376 1164 R   20  0.0   1:32.00 qmail-smtpd                                                                                                                                   
 2497 qmaild    25   0  3900 1376 1164 R   20  0.0   1:19.18 qmail-smtpd                                                                                                                                   
 3180 qmaild    25   0  3784 1380 1164 R   20  0.0   0:38.09 qmail-smtpd                                                                                                                                   
 3232 qmaild    25   0  4192 1380 1164 R   20  0.0   0:33.59 qmail-smtpd                                                                                                                                   
 3599 qmaild    25   0  4064 1380 1164 R   20  0.0   0:15.39 qmail-smtpd                                                                                                                                   
 3691 qmaild    25   0  4256 1384 1164 R   20  0.0   0:11.59 qmail-smtpd                                                                                                                                   
 3760 qmaild    25   0  3476 1376 1164 R   20  0.0   0:07.82 qmail-smtpd                                                                                                                                   
 3822 qmaild    25   0  4772 1376 1164 R   20  0.0   0:04.00 qmail-smtpd                                                                                                                                   
 1934 qmaild    25   0  4564 1376 1164 R   20  0.0   1:42.62 qmail-smtpd                                                                                                                                   
 3097 qmaild    25   0  4560 1384 1164 R   20  0.0   0:43.00 qmail-smtpd                                                                                                                                   
 3253 qmaild    25   0  4964 1376 1164 R   20  0.0   0:34.51 qmail-smtpd                                                                                                                                   
 3695 qmaild    25   0  4428 1376 1164 R   19  0.0   0:11.50 qmail-smtpd                                                                                                                                   
 2360 qmaild    25   0  5268 1376 1164 R   18  0.0   1:23.78 qmail-smtpd                                                                                                                                   
 3018 qmaild    25   0  3536 1380 1164 R   17  0.0   0:47.69 qmail-smtpd                                                                                                                                   
 3234 qmaild    25   0  4732 1380 1164 R   17  0.0   0:34.96 qmail-smtpd                                                                                                                                   
 3859 qmaild    25   0  3736 1376 1164 R   17  0.0   0:02.72 qmail-smtpd                                                                                                                                   
 2089 qmaild    25   0  5012 1380 1164 R   17  0.0   1:33.78 qmail-smtpd                                                                                                                                   
 2538 qmaild    25   0  3904 1384 1164 R   17  0.0   1:16.59 qmail-smtpd                                                                                                                                   
 2565 qmaild    25   0  4208 1380 1164 R   17  0.0   1:15.60 qmail-smtpd                                                                                                                                   
 3101 qmaild    25   0  4424 1388 1164 R   17  0.0   0:41.59 qmail-smtpd                                                                                                                                   
 3318 qmaild    25   0  3584 1384 1164 R   17  0.0   0:30.79 qmail-smtpd                                                                                                                                   
 3375 qmaild    25   0  5224 1380 1164 R   17  0.0   0:26.09 qmail-smtpd                                                                                                                                   
 3491 qmaild    25   0  5200 1380 1164 R   17  0.0   0:21.49 qmail-smtpd                                                                                                                                   
 3525 qmaild    25   0  5264 1380 1164 R   17  0.0   0:19.02 qmail-smtpd                                                                                                                                   
 3680 qmaild    25   0  4452 1380 1164 R   17  0.0   0:11.99 qmail-smtpd                                                                                                                                   
 3765 qmaild    25   0  4532 1376 1164 R   17  0.0   0:07.11 qmail-smtpd                                                                                                                                   
 3829 qmaild    25   0  4128 1380 1164 R   17  0.0   0:03.59 qmail-smtpd                                                                                                                                   
 3866 qmaild    25   0  5028 1376 1164 R   17  0.0   0:02.29 qmail-smtpd                                                                                                                                   
 3725 qmaild    25   0  5272 1380 1164 R   16  0.0   0:10.29 qmail-smtpd                                                                                                                                   
 3885 qmaild    25   0  4560 1380 1164 R   15  0.0   0:01.64 qmail-smtpd                                                                                                                                   
 2984 apache    15   0 53520  25m 3844 S   11  0.6   0:01.45 httpd                                                                                                                                         
19168 apache    15   0 53636  26m 5348 S    1  0.7   0:16.95 httpd                                                                                                                                         
 2979 apache    15   0 53860  25m 3780 S    1  0.6   0:01.69 httpd                                                                                                                                         
 3445 apache    15   0 53548  25m 3924 S    1  0.6   0:01.27 httpd                                                                                                                                         
 3891 root      16   0  2804 1036  764 R    1  0.0   0:00.05 top                                                                                                                                           
 4

In poking around and watching /var/log/secure we can see the following:

Code: Select all

May 28 20:40:20 www xinetd[10610]: FAIL: smtp service_limit from=87.106.66.83
May 28 20:40:21 www xinetd[10610]: FAIL: smtp service_limit from=67.32.139.58
May 28 20:40:22 www xinetd[10610]: FAIL: smtp service_limit from=62.157.100.165
May 28 20:40:22 www xinetd[10610]: FAIL: smtp service_limit from=212.168.16.99
May 28 20:40:22 www xinetd[10610]: START: smtp pid=4094 from=80.68.93.48
May 28 20:40:23 www xinetd[10610]: FAIL: smtp service_limit from=217.172.161.34
May 28 20:40:23 www xinetd[10610]: START: smtp pid=4101 from=89.234.0.219
May 28 20:40:23 www xinetd[10610]: FAIL: smtp service_limit from=210.167.87.95
May 28 20:40:24 www xinetd[10610]: START: smtp pid=4108 from=82.186.102.180
May 28 20:40:24 www xinetd[10610]: FAIL: smtp service_limit from=65.61.200.88
May 28 20:40:25 www xinetd[10610]: START: smtp pid=4115 from=207.44.220.14
May 28 20:40:28 www xinetd[10610]: START: smtp pid=4123 from=69.147.103.224
May 28 20:40:28 www xinetd[10610]: FAIL: smtp service_limit from=190.20.139.217
May 28 20:40:28 www xinetd[10610]: START: smtp pid=4127 from=211.103.110.109
May 28 20:40:28 www xinetd[10610]: FAIL: smtp service_limit from=124.83.61.197
May 28 20:40:29 www xinetd[10610]: START: smtp pid=4137 from=69.7.35.20
May 28 20:40:30 www xinetd[10610]: FAIL: smtp service_limit from=68.142.202.118
May 28 20:40:30 www xinetd[10610]: FAIL: smtp service_limit from=194.140.3.111
May 28 20:40:30 www xinetd[10610]: FAIL: smtp service_limit from=70.169.213.227
May 28 20:40:31 www xinetd[10610]: FAIL: smtp service_limit from=194.112.189.146
May 28 20:40:32 www xinetd[10610]: FAIL: smtp service_limit from=61.207.12.188
May 28 20:40:32 www xinetd[10610]: FAIL: smtp service_limit from=80.146.227.250
May 28 20:40:32 www xinetd[10610]: FAIL: smtp service_limit from=216.196.243.82
May 28 20:40:33 www xinetd[10610]: START: smtp pid=4154 from=154.33.69.56
May 28 20:40:33 www xinetd[10610]: START: smtp pid=4155 from=66.129.74.135
May 28 20:40:33 www xinetd[10610]: FAIL: smtp service_limit from=192.244.211.157
May 28 20:40:33 www xinetd[10610]: FAIL: smtp service_limit from=212.152.145.79
May 28 20:40:34 www xinetd[10610]: FAIL: smtp service_limit from=124.83.170.95
May 28 20:40:34 www xinetd[10610]: FAIL: smtp service_limit from=89.107.160.108
May 28 20:40:34 www xinetd[10610]: FAIL: smtp service_limit from=128.235.251.32
May 28 20:40:34 www xinetd[10610]: FAIL: smtp service_limit from=87.250.129.51
May 28 20:40:34 www xinetd[10610]: START: smtp pid=4162 from=213.221.235.5
May 28 20:40:34 www xinetd[10610]: FAIL: smtp service_limit from=80.98.28.27
May 28 20:40:34 www xinetd[10610]: FAIL: smtp service_limit from=194.112.189.146

We've had qmail going up and down all day. At one point we had our server load hit 1000. That was pretty cool, in a masochistic sort of way. Anyhow, I'm wondering, would installing ASL help against this type of attack?

I'm really "not" a server admin per say, and occasionally stuff like this comes up and I feel a little like that small kid who ended up in the wrong swimming class at the pool.

Short of IPTABLE banning all the offending IP addresses as they repeat (which I would be doing by hand at present) is there any way withing Plesk 8.3 to limit access to Qmail?

tungsten · Unread post by **tungsten** » Wed May 28, 2008 10:00 pm

Hmm... it seems that qmail was actually off these last few hours. That explains why the load was returning to normal.

Since flipping Qmail back on, our load has spooled up to 70 in the last 3 minutes and climbing.

/sigh. It's going to be a long night.

Unread post by **scott** » Wed May 28, 2008 11:12 pm

Theres a max load setting in xinetd as I recall. It will automatically stop accepting connections when it hits a certain limit.

ASL could automatically add/expire firewall rules based on some of the output there. We might need to work up a custom rule for you, but I think we could get that knocked out pretty quickly. I'd need as big of a sample from your logs there as I could get, just to make sure we catch all the conditions.

exi1ed0ne · Unread post by **exi1ed0ne** » Thu May 29, 2008 9:47 am

Adding the following line might help to the smtp_psa xinetd entry:

Code: Select all

per_source	= 1

That will limit the incoming connections to one connection per IP at a time. I've found it cuts down on the load, and legit email isn't impacted. This won't help if you are getting slammed from many places, but it can't hurt.