store | blogs | forums | twitter | facebook | wiki | downloads | support portal
Atomic Secure Linux
It is currently Thu Nov 21, 2019 3:47 pm

» Feed - Atomicorp

All times are UTC - 5 hours [ DST ]




Post new topic Reply to topic  [ 39 posts ]  Go to page 1, 2, 3  Next
Author Message
 Post subject: Extreme incoming Email killing CPu
Unread postPosted: Wed Jul 31, 2013 5:09 am 
Offline
Forum Regular
Forum Regular

Joined: Sat Dec 11, 2004 2:33 pm
Posts: 243
Location: South Africa
Hello,

Over the past couple of weeks, my server has seen a huge increase in incomming email.
Hundreds of connections at a time to server are causing spamd to cosnume massive ammounts of CPU.
System load often runs at 15/20/20 and is causing me all kinds of sissues.

Can anybosy suggest a way to investigate and reduce the load?

Sample of maillog

Jul 30 14:30:23 sa1 /var/qmail/bin/relaylock[126847]: /var/qmail/bin/relaylock: mail from 24.232.235.168:52393 (ol168-235.fibertel.com.ar)
Jul 30 14:30:23 sa1 /var/qmail/bin/relaylock[126780]: /var/qmail/bin/relaylock: mail from 95.56.10.169:10139 (95.56.10.169.megaline.telecom.kz)
Jul 30 14:30:23 sa1 /var/qmail/bin/relaylock[126851]: /var/qmail/bin/relaylock: mail from 91.98.228.214:12577 (91.98.228.214.pol.ir)
Jul 30 14:30:23 sa1 /var/qmail/bin/relaylock[126853]: /var/qmail/bin/relaylock: mail from 201.4.160.220:18176 (201-4-160-220.user.veloxzone.com.br)
Jul 30 14:30:23 sa1 /var/qmail/bin/relaylock[126802]: /var/qmail/bin/relaylock: mail from 41.138.52.126:51642 (not defined)
Jul 30 14:30:23 sa1 /var/qmail/bin/relaylock[126855]: /var/qmail/bin/relaylock: mail from 91.218.221.94:6688 (mail.dokltd.ru)
Jul 30 14:30:23 sa1 /var/qmail/bin/relaylock[126856]: /var/qmail/bin/relaylock: mail from 89.67.28.11:14100 (89-67-28-11.dynamic.chello.pl)
Jul 30 14:30:23 sa1 /var/qmail/bin/relaylock[126858]: /var/qmail/bin/relaylock: mail from 213.139.25.251:2503 (not defined)
Jul 30 14:30:23 sa1 /var/qmail/bin/relaylock[126861]: /var/qmail/bin/relaylock: mail from 202.154.112.180:20698 (mail.howas.com.au)
Jul 30 14:30:23 sa1 /var/qmail/bin/relaylock[126860]: /var/qmail/bin/relaylock: mail from 111.91.234.78:24684 (not defined)

_________________
Mark Brindley
2Large Networks - Web solutions that work


Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Wed Jul 31, 2013 8:44 am 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin

Joined: Wed Dec 31, 1969 8:00 pm
Posts: 8329
Location: earth
From the same IP's? Or is this like a botnet spamming you?

Greylisting (yum install qgreylist) could help there, if its common-ish IP's you can limit the max connections per IP in xinetd.


Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Wed Jul 31, 2013 9:34 am 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Thu Dec 09, 2004 11:19 am
Posts: 2321
This is definitely a spam botnet. I recognise the "signature" from the rdns.

You will stop a huge number if you geoblock south America (Argentina, Brazil, Chile, Pero, mostly) if you can do so without impact to your customers. Geoblocking Eastern Europe is also sensible (Russia, Romania, Ukraine mostly) and of course China and South Korea. India is becoming a problem too. But geoblocking isn't a solution for everybody, and doesn't solve the problem completely anyway.

Another option, as Scott says, is to set a load level above which email will be temporarily rejected in xinetd.

However, you have other options too, and the key is to get a balance.

High load (both CPU and memory) is caused by spamassassin and clamav checking the emails. Clamav uses a hell of a lot of memory, for example.

If left at the default, qmail will accept an unlimited number of connections at once, causing massive memory use, high CPU use, and generally if you get too many emails coming in at once everything falls over.

You can cure this instantly by limiting the number of simultaneous connections, again in xinetd for smtp_psa.
You'd be surprised at how low you can go, even for a busy server, without causing noticeable email delays. I have some of mine set as low as 5, but 10 seems optimal.

Example lines to add to smtp_psa (position doesn't really matter as long as it is between the { and })

Code:
        instances       = 25
        per_source      = 5
        max_load        = 15


instances = how many simultaneous connections
per_source = how many connections from an individual IP
max_load = load level above which to stop accepting incoming email



Next, look at no-load/low-load blocking - blocking before sa and clamav get hold of the message. with spamdyke, for example, you might add a 5 to 10 second answer delay, enable graylisting (though we don't bother anymore), and possibly add some more rbls to your arsenal.

Rejecting messages from IPs with no rDNS is a MUST if you don't already do so. You won't get many false positives on that and spamdyke excels as this task.

Note that the majority of the botnet traffic comes from domestic broadband connections, which spamdyke can detect and block based on their rDNS pattern. Unfortunately we've found too many businesses setting up their own mailservers on such connections, not bothering to get the rdns changed to something more appropriate than the default, or not being able to. The result is too many false positives.

Another thing you may notice if you look at the logs in any detail is that a significant proportion of the botnet spam is being sent to non-existent addresses - usually obviously non-existent ones (random hex strings), or existing ones with some additional characters added (most often a "d"). If I were able to create my own MTA, I would have it check for a valid address before any av or spam check if the incoming message it were to pass initial rbl/rdns/etc checks. I don't think this is possible, even with postfix, unfortunately.

_________________
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>


Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Wed Jul 31, 2013 10:12 am 
Offline
Forum Regular
Forum Regular

Joined: Sat Dec 11, 2004 2:33 pm
Posts: 243
Location: South Africa
Hi Scott,

It is 1000's of unique IP's
I have already set these values in /etc/xinetd.d/smtp_psa

max_load = 10
per_source = 5
instances = 50

I use Spamdyke and have just enabled log-level=debug
I can see 1000's of entries in the logs for those connection as EARLYTALKERS.

Jul 31 15:51:51 sa1 spamdyke[73754]: FILTER_EARLYTALKER delay: 4
Jul 31 15:51:51 sa1 /var/qmail/bin/relaylock[73758]: /var/qmail/bin/relaylock: mail from 190.216.230.135:45411 (not defined)
Jul 31 15:51:51 sa1 /var/qmail/bin/relaylock[73759]: /var/qmail/bin/relaylock: mail from 173.8.142.97:47389 (173-8-142-97-sfba.hfc.comcastbusiness.net)
Jul 31 15:51:51 sa1 spamdyke[73758]: DEBUG(filter_earlytalker()@filter.c:1856): checking for earlytalker; delay: 4


Should I try qgreylist or just use the Spamdyke Grey list?

_________________
Mark Brindley
2Large Networks - Web solutions that work


Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Wed Jul 31, 2013 10:37 am 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin

Joined: Wed Dec 31, 1969 8:00 pm
Posts: 8329
Location: earth
we use qgreylist on our edge Project Gamera servers. Those do millions of messages a day... my only observation on it is I wrote that package 8 or 9 years ago and havent had to touch it since :P

rbls might make even more of a difference, since it happens first (or layer it.). Normally I do all my rbl processing inside of spamassassin to help train the spam bayes db. In your case you probably just need to get your throttling under control.


Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Wed Jul 31, 2013 10:43 am 
Offline
Forum Regular
Forum Regular

Joined: Sat Dec 11, 2004 2:33 pm
Posts: 243
Location: South Africa
Hiya Faris,

Thanks for the in-depth reply.

Quote:
You will stop a huge number if you geo block south America (Argentina, Brazil, Chile, Pero, mostly) if you can do so without impact to your customers. Geoblocking Eastern Europe is also sensible (Russia, Romania, Ukraine mostly) and of course China and South Korea.


I have added these countries and a bunch more to the ASL GEO Blocking, but i still see a number of IP's connecting from many of the countries. Is this the correct place to block them?

Quote:
Another option, as Scott says, is to set a load level above which email will be temporarily rejected in xinetd.

I have the following values in /etc/xinetd.d/smtp_psa
max_load = 10
per_source = 5
instances = 50
I see alot of these messages in the ASL log
Deactivating service smtp due to excessive incoming connections. Restarting in 10 seconds

Quote:
High load (both CPU and memory) is caused by spamassassin and clamav checking the emails. Clamav uses a hell of a lot of memory, for example.

Yes I see this every day, drives me bonkers!

Quote:
you might add a 5 to 10 second answer delay

greeting-delay-secs=4

Quote:
possibly add some more rbls to your arsenal.

I am using the following RBLs, can you suggest any other effective ones?
dns-blacklist-entry=zen.spamhaus.org
dns-blacklist-entry=bl.spamcop.net
dns-blacklist-entry=b.barracudacentral.org
dns-blacklist-entry=db.rurbl.ru

Quote:
Rejecting messages from IPs with no rDNS is a MUST if you don't already do so.

I have the following setting for Spamdyke
reject-missing-sender-mx
reject-empty-rdns
reject-unresolvable-rdns

_________________
Mark Brindley
2Large Networks - Web solutions that work


Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Wed Jul 31, 2013 11:02 am 
Offline
Forum Regular
Forum Regular

Joined: Sat Dec 11, 2004 2:33 pm
Posts: 243
Location: South Africa
Hi Scott,

Quote:
we use qgreylist on our edge Project Gamera servers. Those do millions of messages a day... my only observation on it is I wrote that package 8 or 9 years ago and havent had to touch it since :P

I installed qgreylist, i still see tons of connections.
I looked in /var/qmail/greylist and don't see any ips orl ogs there.
Do I need to add anything to xinetd for smtp_psa?

Quote:
rbls might make even more of a difference, since it happens first (or layer it.). Normally I do all my rbl processing inside of spamassassin to help train the spam bayes db.

Could you possibly suggest a couple fo Good rbls?
Where can I grab some more information on processing RBL inside of spamassassin?

_________________
Mark Brindley
2Large Networks - Web solutions that work


Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Wed Jul 31, 2013 2:53 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin

Joined: Wed Dec 31, 1969 8:00 pm
Posts: 8329
Location: earth
did you restart xinetd after installing qgreylist?


Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Thu Aug 01, 2013 4:06 am 
Offline
Forum Regular
Forum Regular

Joined: Sat Dec 11, 2004 2:33 pm
Posts: 243
Location: South Africa
Hello Scott,

Yeah, sorry I was a bit quick with my reply.
After restarting xinetd greylisting started working.

Greylisting started off really well, blocking tons of connectsions.
IP x.x.x.x back too soon - temp error again
IP x.x.x.x new - temp error

after a couple of minutes, 1000's of the bad ip's are now been aceepted.
IP 37.237.33.17 OK - accepting

if I tail the log and grep accepting the log runs like a open tap.
here is a quick jing of the log file http://www.2large.co.za/jing/spam-maillog.swf

any suggestions will be great.

_________________
Mark Brindley
2Large Networks - Web solutions that work


Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Thu Aug 01, 2013 6:41 am 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Thu Dec 09, 2004 11:19 am
Posts: 2321
instances = 50 could be way too high. Try 25 or even lower.

earlytalkers - you can set the timeout to 6 or even a tad higher. 4 is a bit low in my experience.

If you already have spamdyke installed, why not use it for graylisting instead of adding an additional package?

The geoblocking in ASL will only normally work correctly if you only have the ASL firewall active -- and not any other firewall (e.g. Plesk firewall, apf etc)

_________________
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>


Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Thu Aug 01, 2013 7:23 am 
Offline
Forum Regular
Forum Regular

Joined: Sat Dec 11, 2004 2:33 pm
Posts: 243
Location: South Africa
Hi faris,

Quote:
instances = 50 could be way too high. Try 25 or even lower.

I have lowered it to 25

Quote:
earlytalkers - you can set the timeout to 6 or even a tad higher. 4 is a bit low in my experience.

I had to drop it to 3, after watching the log is noticed a number of local service providers timing out.

Quote:
The geoblocking in ASL will only normally work correctly if you only have the ASL firewall active -- and not any other firewall (e.g. Plesk firewall, apf etc)

I have stoped psa-firewall but i am still seeying connection from countries that i have blocked!
Any suggestions on a better way to test will be appreciated.

I am considering setting up another server to run Project Gamera to handle the load and pass clean mail onto the PSA server.
Has anybody sucessfully implmented this?
I am not qute sure how to go about it, any tips on the setup would be great.

_________________
Mark Brindley
2Large Networks - Web solutions that work


Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Thu Aug 01, 2013 9:00 am 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin

Joined: Wed Dec 31, 1969 8:00 pm
Posts: 8329
Location: earth
Yeah thats how we handle mail here. We have a group of PG servers that handle all our inbound smtp traffic. Its an SMTP smarthost, you set your MX records to point to the PG systems, and they in turn relay traffic to the downstream servers.


Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Fri Aug 02, 2013 8:27 am 
Offline
Forum Regular
Forum Regular

Joined: Sat Dec 11, 2004 2:33 pm
Posts: 243
Location: South Africa
hello,

Today is the worst day so far, server is getting smashed!
System load hit 75. and finally fell over :)

Clients are ranting (as they do) when mail delivery is slow and delayed.

Is there a way to block port 25 on PSA?
Will this slow things down?

Server is running smtps_ and submission 587
Will simpler to get contact clients and get them to change SMTP ports.

_________________
Mark Brindley
2Large Networks - Web solutions that work


Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Fri Aug 02, 2013 5:16 pm 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Thu Dec 09, 2004 11:19 am
Posts: 2321
If you have max_load = 10 the systeml should have stopped accepting incoming connections and the load should not have gone much higher.

And with the other settings you have, I really would not expect the load to have got to 25

Could something else be happening on top of the email situation?

_________________
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>


Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Mon Aug 05, 2013 1:37 pm 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Thu Dec 09, 2004 11:19 am
Posts: 2321
Oh boy oh boy oh boy.

I've just logged in and had a look at kram's problem. I've never seen anything like it.

There are connections coming in on port 25 at an astronomical rate. I mean utterly massive, huge, mega, incredible rate. I can see no pattern in the connecting IPs - each one seems to be different.

The maillog and /var/log/secure /var/log/messages don't tell me anything at all. All you see if a relaylock entry with the connecting IP in the maillog. No "from", no "to", no "rejected", no "failed", no nothing at all.

When adding a greeting-timeout of just 2 seconds in spamdyke, each and every one of those connections hits this filter, implying to me that these connections are earlytalkers and just throwing data at the server.

This may explain why, when using tcpdump on port 25, 99% of what I see is noise -- in 20Mb of capture, all I see is an average of three "real" email conversations - the rest of it is "truncated" or "retransmitted" or basically gibberish and there's no data for me to work with. At least none that I can extract using wireshark.

Does anyone have any more suggestions? Kram is basically being hit by millions of connections from every IP under the sun, and it is as though no valid smtp conversation is taking place.

I know I'm missing some trick here. There must be something more I can do to try and find out if all this traffic is directed towards one particular domain or IP, maybe?

Suggestions please -- this is not only driving him insane, it is now driving me insane because I can't get to the bottom of it.

It is like a port 25 DDoS.

_________________
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 39 posts ]  Go to page 1, 2, 3  Next

» Feed - Atomicorp

All times are UTC - 5 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group