Extreme incoming Email killing CPu

Forum for getting help with Project Gamera, Spamassassin, Clamav, qmail-scanner and other anti-spam tools.
kram
Forum Regular
Forum Regular
Posts: 243
Joined: Sat Dec 11, 2004 2:33 pm
Location: South Africa

Extreme incoming Email killing CPu

Unread post by kram »

Hello,

Over the past couple of weeks, my server has seen a huge increase in incomming email.
Hundreds of connections at a time to server are causing spamd to cosnume massive ammounts of CPU.
System load often runs at 15/20/20 and is causing me all kinds of sissues.

Can anybosy suggest a way to investigate and reduce the load?

Sample of maillog

Jul 30 14:30:23 sa1 /var/qmail/bin/relaylock[126847]: /var/qmail/bin/relaylock: mail from 24.232.235.168:52393 (ol168-235.fibertel.com.ar)
Jul 30 14:30:23 sa1 /var/qmail/bin/relaylock[126780]: /var/qmail/bin/relaylock: mail from 95.56.10.169:10139 (95.56.10.169.megaline.telecom.kz)
Jul 30 14:30:23 sa1 /var/qmail/bin/relaylock[126851]: /var/qmail/bin/relaylock: mail from 91.98.228.214:12577 (91.98.228.214.pol.ir)
Jul 30 14:30:23 sa1 /var/qmail/bin/relaylock[126853]: /var/qmail/bin/relaylock: mail from 201.4.160.220:18176 (201-4-160-220.user.veloxzone.com.br)
Jul 30 14:30:23 sa1 /var/qmail/bin/relaylock[126802]: /var/qmail/bin/relaylock: mail from 41.138.52.126:51642 (not defined)
Jul 30 14:30:23 sa1 /var/qmail/bin/relaylock[126855]: /var/qmail/bin/relaylock: mail from 91.218.221.94:6688 (mail.dokltd.ru)
Jul 30 14:30:23 sa1 /var/qmail/bin/relaylock[126856]: /var/qmail/bin/relaylock: mail from 89.67.28.11:14100 (89-67-28-11.dynamic.chello.pl)
Jul 30 14:30:23 sa1 /var/qmail/bin/relaylock[126858]: /var/qmail/bin/relaylock: mail from 213.139.25.251:2503 (not defined)
Jul 30 14:30:23 sa1 /var/qmail/bin/relaylock[126861]: /var/qmail/bin/relaylock: mail from 202.154.112.180:20698 (mail.howas.com.au)
Jul 30 14:30:23 sa1 /var/qmail/bin/relaylock[126860]: /var/qmail/bin/relaylock: mail from 111.91.234.78:24684 (not defined)
Mark Brindley
2Large Networks - Web solutions that work
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8330
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Re: Extreme incoming Email killing CPu

Unread post by scott »

From the same IP's? Or is this like a botnet spamming you?

Greylisting (yum install qgreylist) could help there, if its common-ish IP's you can limit the max connections per IP in xinetd.
faris
Long Time Forum Regular
Long Time Forum Regular
Posts: 2321
Joined: Thu Dec 09, 2004 11:19 am

Re: Extreme incoming Email killing CPu

Unread post by faris »

This is definitely a spam botnet. I recognise the "signature" from the rdns.

You will stop a huge number if you geoblock south America (Argentina, Brazil, Chile, Pero, mostly) if you can do so without impact to your customers. Geoblocking Eastern Europe is also sensible (Russia, Romania, Ukraine mostly) and of course China and South Korea. India is becoming a problem too. But geoblocking isn't a solution for everybody, and doesn't solve the problem completely anyway.

Another option, as Scott says, is to set a load level above which email will be temporarily rejected in xinetd.

However, you have other options too, and the key is to get a balance.

High load (both CPU and memory) is caused by spamassassin and clamav checking the emails. Clamav uses a hell of a lot of memory, for example.

If left at the default, qmail will accept an unlimited number of connections at once, causing massive memory use, high CPU use, and generally if you get too many emails coming in at once everything falls over.

You can cure this instantly by limiting the number of simultaneous connections, again in xinetd for smtp_psa.
You'd be surprised at how low you can go, even for a busy server, without causing noticeable email delays. I have some of mine set as low as 5, but 10 seems optimal.

Example lines to add to smtp_psa (position doesn't really matter as long as it is between the { and })

Code: Select all

        instances       = 25
        per_source      = 5
        max_load        = 15
instances = how many simultaneous connections
per_source = how many connections from an individual IP
max_load = load level above which to stop accepting incoming email



Next, look at no-load/low-load blocking - blocking before sa and clamav get hold of the message. with spamdyke, for example, you might add a 5 to 10 second answer delay, enable graylisting (though we don't bother anymore), and possibly add some more rbls to your arsenal.

Rejecting messages from IPs with no rDNS is a MUST if you don't already do so. You won't get many false positives on that and spamdyke excels as this task.

Note that the majority of the botnet traffic comes from domestic broadband connections, which spamdyke can detect and block based on their rDNS pattern. Unfortunately we've found too many businesses setting up their own mailservers on such connections, not bothering to get the rdns changed to something more appropriate than the default, or not being able to. The result is too many false positives.

Another thing you may notice if you look at the logs in any detail is that a significant proportion of the botnet spam is being sent to non-existent addresses - usually obviously non-existent ones (random hex strings), or existing ones with some additional characters added (most often a "d"). If I were able to create my own MTA, I would have it check for a valid address before any av or spam check if the incoming message it were to pass initial rbl/rdns/etc checks. I don't think this is possible, even with postfix, unfortunately.
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
kram
Forum Regular
Forum Regular
Posts: 243
Joined: Sat Dec 11, 2004 2:33 pm
Location: South Africa

Re: Extreme incoming Email killing CPu

Unread post by kram »

Hi Scott,

It is 1000's of unique IP's
I have already set these values in /etc/xinetd.d/smtp_psa

max_load = 10
per_source = 5
instances = 50

I use Spamdyke and have just enabled log-level=debug
I can see 1000's of entries in the logs for those connection as EARLYTALKERS.

Jul 31 15:51:51 sa1 spamdyke[73754]: FILTER_EARLYTALKER delay: 4
Jul 31 15:51:51 sa1 /var/qmail/bin/relaylock[73758]: /var/qmail/bin/relaylock: mail from 190.216.230.135:45411 (not defined)
Jul 31 15:51:51 sa1 /var/qmail/bin/relaylock[73759]: /var/qmail/bin/relaylock: mail from 173.8.142.97:47389 (173-8-142-97-sfba.hfc.comcastbusiness.net)
Jul 31 15:51:51 sa1 spamdyke[73758]: DEBUG(filter_earlytalker()@filter.c:1856): checking for earlytalker; delay: 4


Should I try qgreylist or just use the Spamdyke Grey list?
Mark Brindley
2Large Networks - Web solutions that work
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8330
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Re: Extreme incoming Email killing CPu

Unread post by scott »

we use qgreylist on our edge Project Gamera servers. Those do millions of messages a day... my only observation on it is I wrote that package 8 or 9 years ago and havent had to touch it since :P

rbls might make even more of a difference, since it happens first (or layer it.). Normally I do all my rbl processing inside of spamassassin to help train the spam bayes db. In your case you probably just need to get your throttling under control.
kram
Forum Regular
Forum Regular
Posts: 243
Joined: Sat Dec 11, 2004 2:33 pm
Location: South Africa

Re: Extreme incoming Email killing CPu

Unread post by kram »

Hiya Faris,

Thanks for the in-depth reply.
You will stop a huge number if you geo block south America (Argentina, Brazil, Chile, Pero, mostly) if you can do so without impact to your customers. Geoblocking Eastern Europe is also sensible (Russia, Romania, Ukraine mostly) and of course China and South Korea.
I have added these countries and a bunch more to the ASL GEO Blocking, but i still see a number of IP's connecting from many of the countries. Is this the correct place to block them?
Another option, as Scott says, is to set a load level above which email will be temporarily rejected in xinetd.
I have the following values in /etc/xinetd.d/smtp_psa
max_load = 10
per_source = 5
instances = 50
I see alot of these messages in the ASL log
Deactivating service smtp due to excessive incoming connections. Restarting in 10 seconds
High load (both CPU and memory) is caused by spamassassin and clamav checking the emails. Clamav uses a hell of a lot of memory, for example.
Yes I see this every day, drives me bonkers!
you might add a 5 to 10 second answer delay
greeting-delay-secs=4
possibly add some more rbls to your arsenal.
I am using the following RBLs, can you suggest any other effective ones?
dns-blacklist-entry=zen.spamhaus.org
dns-blacklist-entry=bl.spamcop.net
dns-blacklist-entry=b.barracudacentral.org
dns-blacklist-entry=db.rurbl.ru
Rejecting messages from IPs with no rDNS is a MUST if you don't already do so.
I have the following setting for Spamdyke
reject-missing-sender-mx
reject-empty-rdns
reject-unresolvable-rdns
Mark Brindley
2Large Networks - Web solutions that work
kram
Forum Regular
Forum Regular
Posts: 243
Joined: Sat Dec 11, 2004 2:33 pm
Location: South Africa

Re: Extreme incoming Email killing CPu

Unread post by kram »

Hi Scott,
we use qgreylist on our edge Project Gamera servers. Those do millions of messages a day... my only observation on it is I wrote that package 8 or 9 years ago and havent had to touch it since :P
I installed qgreylist, i still see tons of connections.
I looked in /var/qmail/greylist and don't see any ips orl ogs there.
Do I need to add anything to xinetd for smtp_psa?
rbls might make even more of a difference, since it happens first (or layer it.). Normally I do all my rbl processing inside of spamassassin to help train the spam bayes db.
Could you possibly suggest a couple fo Good rbls?
Where can I grab some more information on processing RBL inside of spamassassin?
Mark Brindley
2Large Networks - Web solutions that work
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8330
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Re: Extreme incoming Email killing CPu

Unread post by scott »

did you restart xinetd after installing qgreylist?
kram
Forum Regular
Forum Regular
Posts: 243
Joined: Sat Dec 11, 2004 2:33 pm
Location: South Africa

Re: Extreme incoming Email killing CPu

Unread post by kram »

Hello Scott,

Yeah, sorry I was a bit quick with my reply.
After restarting xinetd greylisting started working.

Greylisting started off really well, blocking tons of connectsions.
IP x.x.x.x back too soon - temp error again
IP x.x.x.x new - temp error

after a couple of minutes, 1000's of the bad ip's are now been aceepted.
IP 37.237.33.17 OK - accepting

if I tail the log and grep accepting the log runs like a open tap.
here is a quick jing of the log file http://www.2large.co.za/jing/spam-maillog.swf

any suggestions will be great.
Mark Brindley
2Large Networks - Web solutions that work
faris
Long Time Forum Regular
Long Time Forum Regular
Posts: 2321
Joined: Thu Dec 09, 2004 11:19 am

Re: Extreme incoming Email killing CPu

Unread post by faris »

instances = 50 could be way too high. Try 25 or even lower.

earlytalkers - you can set the timeout to 6 or even a tad higher. 4 is a bit low in my experience.

If you already have spamdyke installed, why not use it for graylisting instead of adding an additional package?

The geoblocking in ASL will only normally work correctly if you only have the ASL firewall active -- and not any other firewall (e.g. Plesk firewall, apf etc)
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
kram
Forum Regular
Forum Regular
Posts: 243
Joined: Sat Dec 11, 2004 2:33 pm
Location: South Africa

Re: Extreme incoming Email killing CPu

Unread post by kram »

Hi faris,
instances = 50 could be way too high. Try 25 or even lower.
I have lowered it to 25
earlytalkers - you can set the timeout to 6 or even a tad higher. 4 is a bit low in my experience.
I had to drop it to 3, after watching the log is noticed a number of local service providers timing out.
The geoblocking in ASL will only normally work correctly if you only have the ASL firewall active -- and not any other firewall (e.g. Plesk firewall, apf etc)
I have stoped psa-firewall but i am still seeying connection from countries that i have blocked!
Any suggestions on a better way to test will be appreciated.

I am considering setting up another server to run Project Gamera to handle the load and pass clean mail onto the PSA server.
Has anybody sucessfully implmented this?
I am not qute sure how to go about it, any tips on the setup would be great.
Mark Brindley
2Large Networks - Web solutions that work
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8330
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Re: Extreme incoming Email killing CPu

Unread post by scott »

Yeah thats how we handle mail here. We have a group of PG servers that handle all our inbound smtp traffic. Its an SMTP smarthost, you set your MX records to point to the PG systems, and they in turn relay traffic to the downstream servers.
kram
Forum Regular
Forum Regular
Posts: 243
Joined: Sat Dec 11, 2004 2:33 pm
Location: South Africa

Re: Extreme incoming Email killing CPu

Unread post by kram »

hello,

Today is the worst day so far, server is getting smashed!
System load hit 75. and finally fell over :)

Clients are ranting (as they do) when mail delivery is slow and delayed.

Is there a way to block port 25 on PSA?
Will this slow things down?

Server is running smtps_ and submission 587
Will simpler to get contact clients and get them to change SMTP ports.
Mark Brindley
2Large Networks - Web solutions that work
faris
Long Time Forum Regular
Long Time Forum Regular
Posts: 2321
Joined: Thu Dec 09, 2004 11:19 am

Re: Extreme incoming Email killing CPu

Unread post by faris »

If you have max_load = 10 the systeml should have stopped accepting incoming connections and the load should not have gone much higher.

And with the other settings you have, I really would not expect the load to have got to 25

Could something else be happening on top of the email situation?
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
faris
Long Time Forum Regular
Long Time Forum Regular
Posts: 2321
Joined: Thu Dec 09, 2004 11:19 am

Re: Extreme incoming Email killing CPu

Unread post by faris »

Oh boy oh boy oh boy.

I've just logged in and had a look at kram's problem. I've never seen anything like it.

There are connections coming in on port 25 at an astronomical rate. I mean utterly massive, huge, mega, incredible rate. I can see no pattern in the connecting IPs - each one seems to be different.

The maillog and /var/log/secure /var/log/messages don't tell me anything at all. All you see if a relaylock entry with the connecting IP in the maillog. No "from", no "to", no "rejected", no "failed", no nothing at all.

When adding a greeting-timeout of just 2 seconds in spamdyke, each and every one of those connections hits this filter, implying to me that these connections are earlytalkers and just throwing data at the server.

This may explain why, when using tcpdump on port 25, 99% of what I see is noise -- in 20Mb of capture, all I see is an average of three "real" email conversations - the rest of it is "truncated" or "retransmitted" or basically gibberish and there's no data for me to work with. At least none that I can extract using wireshark.

Does anyone have any more suggestions? Kram is basically being hit by millions of connections from every IP under the sun, and it is as though no valid smtp conversation is taking place.

I know I'm missing some trick here. There must be something more I can do to try and find out if all this traffic is directed towards one particular domain or IP, maybe?

Suggestions please -- this is not only driving him insane, it is now driving me insane because I can't get to the bottom of it.

It is like a port 25 DDoS.
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
Post Reply