Extreme incoming Email killing CPu

Forum for getting help with Project Gamera, Spamassassin, Clamav, qmail-scanner and other anti-spam tools.
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8355
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Re: Extreme incoming Email killing CPu

Unread post by scott »

What do the packets look like? Just a SYN or a full 3-way handshake?
kram
Forum Regular
Forum Regular
Posts: 243
Joined: Sat Dec 11, 2004 2:33 pm
Location: South Africa

Re: Extreme incoming Email killing CPu

Unread post by kram »

@faris

Thank you so much for taking the time to look.
I really appreciate all your time and effort.

I really hope somebody else here will be able to shed some light.
My clients are starting to rant and long standing clients are also threatening to leave :(

As a temp measure I have started moving important clients over to a new PSA 11.5 server.
Just hope the problem does not migrate as well.
Mark Brindley
2Large Networks - Web solutions that work
faris
Long Time Forum Regular
Long Time Forum Regular
Posts: 2321
Joined: Thu Dec 09, 2004 11:19 am

Re: Extreme incoming Email killing CPu

Unread post by faris »

There's more than just a SYN, but...this is outside my knowledge to track

Here's a couple of screen shots of a fragment.
Attachments
fragment 2
fragment 2
ws1.jpg (232.31 KiB) Viewed 14491 times
wireshark fragment
wireshark fragment
ws2a.jpg (246 KiB) Viewed 14491 times
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
faris
Long Time Forum Regular
Long Time Forum Regular
Posts: 2321
Joined: Thu Dec 09, 2004 11:19 am

Re: Extreme incoming Email killing CPu

Unread post by faris »

Just to emphasise this, if I search for RCPT TO or FROM in one of the 20Mb captures (just port 25, remember), I get two or three hits, no more, and these seem to be part of a full conversation. The rest is all this "noise".
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
faris
Long Time Forum Regular
Long Time Forum Regular
Posts: 2321
Joined: Thu Dec 09, 2004 11:19 am

Re: Extreme incoming Email killing CPu

Unread post by faris »

Oh. I noticed I was sorting by IP in the screen shots I last sent.

Here's one ordered by time.
Attachments
ws3
ws3
ws3a.jpg (241.18 KiB) Viewed 14487 times
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
faris
Long Time Forum Regular
Long Time Forum Regular
Posts: 2321
Joined: Thu Dec 09, 2004 11:19 am

Re: Extreme incoming Email killing CPu

Unread post by faris »

OK, OK, so what I've been posting has probably been incomprehensible. Blame it on a solid afternoon being deluged with data. I've totally simplified it and hopefully someone can help with this now.

GOOD.JPG
GOOD.JPG
GOOD.JPG
good.jpg (53.26 KiB) Viewed 14484 times
Above is a screen shot of what I expect to see in terms of an SMTP conversation (from a spammer). It is an attempt to relay using the server in question. It follows the form I expect: server says 220, remote issues HELO, server says 250, remote gives RCTP TO, server says 250 and so on and so forth..

I note, incidentally, than in this type of situation there's no domain local to the server being mentioned, so we have no new data to work from.


BAD.JPG
BAD.JPG
BAD.JPG
bad.jpg (119.31 KiB) Viewed 14484 times
Above is screen shot of an example of what happens with these "bad" connections that are driving is nuts.

** They all result in an "Unimplemented" response from the server. All of them. Every single one that I've checked.

You'll see now what I've talking about in terms of "noise". The third line down (31933) contains a big packet of...I don't know what. It isn't until you get three lines from the bottom before the server says 220 and then immediately follows it up with a 502 "Unimplemented".

This is what occurs in each of these thousands of connections that I've looked at.

Is this confirming what I thought? That the spambot is sending data without waiting (hence the earlytalker filter I mentioned before kicking in), so what we're seeing in line 31933 is actually somewhere in the middle of the email, maybe a fragment of its contents, with any reference to a RCPT TO or MAIL FROM long gone?

In these screen shots, I'm searching for conversations by IP address. So this is the "complete" capture of the conversation for this IP (though I note it says "[truncated]" for the command line, presumably because it is long).

And if so.....does this mean we're stuffed in terms of trying to figure out if there's a specific domain being targeted?
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8355
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Re: Extreme incoming Email killing CPu

Unread post by scott »

To help make more sense of this, check out the decoder in wireshark, look under Analyze, Decode TCP stream. Select one of the sessions that dont make any sense and run that against it. It will reassemble the whole session in a more readable human output.

Im just speculating here, but that looks like its encrypted, they're probably trying to do a starttls and its not compatible with the MTA, so thats why you're seeing the 502 error.
faris
Long Time Forum Regular
Long Time Forum Regular
Posts: 2321
Joined: Thu Dec 09, 2004 11:19 am

Re: Extreme incoming Email killing CPu

Unread post by faris »

If I'm doing this correctly, they all look something like this: noise > server response > one or more unimplemented responses

e.g.

Code: Select all

.v=d..D..@".^A.d.N.dk..e..;.@
"e.."eF........n..o..eo.6..J.e.. f.. fSp9f.T.....f.I....Of..4.#.g.3..f....0..f.k7g..Pg...gS.).y..g...gF.>..FX...8h...g.5hh*..h.}.h.g.h..<...<...o...3..1...C.jZ>.iu
{.1>{..p{...{...{...{...{...{...{.."|.."|(.<|(.<|5.U|O`.|v..|...|.^.}.^.}.^.}.*:}..l}.\.}.\.}.\.}1..}...}#..}...~,'8~9.Q~F.j~

220 server-domain.tld ESMTP

502 unimplemented (#5.5.1)

502 unimplemented (#5.5.1)

502 unimplemented (#5.5.1)

502 unimplemented (#5.5.1)

502 unimplemented (#5.5.1)
As far as I can tell, the server in question does support TLS (passthrough from spamdyke to qmail) and I specifically remember seeing a tls encrypted message being received and processed correctly when I was staring at the logs. I also tried doing tcpdump with tls deliberately disabled and they look the same so .... I don't know.

Here's a normal (refused spam) one:

Code: Select all

220 server-hostname.tld ESMTP
EHLO [205.185.139.64]
250-server-hostname.tld
250-AUTH=LOGIN CRAM-MD5 PLAIN
250-AUTH LOGIN CRAM-MD5 PLAIN
250-STARTTLS
250-PIPELINING
250 8BITMIME
MAIL FROM:<name@some-other-domain.tld>
RCPT TO:<name@some-domain.tld>
DATA
250 ok
421 Refused. You have no reverse DNS entry. 
421 Refused. You have no reverse DNS entry. 
QUIT
221 Refused. You have no reverse DNS entry. 
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
BruceLee
Forum Regular
Forum Regular
Posts: 879
Joined: Sat Mar 28, 2009 6:58 pm
Location: Germany

Re: Extreme incoming Email killing CPu

Unread post by BruceLee »

I know thats not the solution and totally not good but it is a strange behaviour.
What happens if you disable TLS for a moment by setting this in spamdyke conf:
tls-level:none
none: Do not provide or allow TLS, even if qmail supports it. qmail's attempt to advertise its TLS support will be hidden and the remote server's request for TLS will be denied.
http://www.spamdyke.org/documentation/README.html#TLS

of course this will cause some trouble for clients using tls but might be better than the ongoing flooding.
or at least it makes it possible to track it down a little bit.
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8355
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Re: Extreme incoming Email killing CPu

Unread post by scott »

The client certainly isnt doing any kind of error checking there, its just dumping everythng to MTA without waiting for it. On the plus side it means this douchebags spam client doesnt actually work, which could maybe be exploited as a way to test for spammer clients. Like greylisting, you throw back some kind of error that a smarter MUA would retry on.
faris
Long Time Forum Regular
Long Time Forum Regular
Posts: 2321
Joined: Thu Dec 09, 2004 11:19 am

Re: Extreme incoming Email killing CPu

Unread post by faris »

I have tried disabling TLS completely and it makes no difference at all -- the traffic continues and the logs are the same.

So, basically broken spambot, then? But spamdyke can block them using a 2 second timeout. Even without it, qmail doesn't give them the time of day.
The thing is, the load is killing the server, even though it doesn't get past qmails greeting > unimplemented.

That's what I want to mitigate. I just don't have any ideas how. If it was directed at a particular domain, we would kill that domain off/move it. But we don't have any way to find out, because the spambot has got well past that stage before any data gets captures and qmail responds.
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8355
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Re: Extreme incoming Email killing CPu

Unread post by scott »

Is there anything in the logs that tells you when its a spambot client? Where Im going with this is an rule that detects the client IP and sets a shun on it.
faris
Long Time Forum Regular
Long Time Forum Regular
Posts: 2321
Joined: Thu Dec 09, 2004 11:19 am

Re: Extreme incoming Email killing CPu

Unread post by faris »

Yes and no. If I set spamdyke logging up one notch, there's a clear FILTER: EARLYTALKER (or similar - I forget the exact format), but the offending IP is not logged on that line.

I would need to see if Sam (Mr Spamdyke, if he'll forgive me for using that term) would help me patch the qmail code to add the offending IP to the log entry.

I am concerned, however, than even if we manage to do this, the sheer number of blocked IPs may result in 1000s of iptables entries. There's plenty of RAM in the system ...8Gb, 16Gb --Kram? I can't remember and I think it has a reasonable amount free to play with.
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8355
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Re: Extreme incoming Email killing CPu

Unread post by scott »

Ive definitely got a solution for large ip lists (called ipsets), in testing Ive done millions of IP's in a single rule. No impact on either memory or performance (looks prettier too!). Its part of the xtables-addon package in the ASL kernels, for non-ASL kernels we could probably do a few hundred thousand the old way without much impact.
kram
Forum Regular
Forum Regular
Posts: 243
Joined: Sat Dec 11, 2004 2:33 pm
Location: South Africa

Re: Extreme incoming Email killing CPu

Unread post by kram »

Hi Faris,

Here is the output from free

Code: Select all

total       used       free     shared    buffers     cached
Mem:      16074312   15630168     444144          0     485564    5633296
-/+ buffers/cache:    9511308    6563004
Swap:      4198972     746540    3452432
Mail delivery is totally out of whack now, mail is coming in 5 hours late.
Connections to the server seem to be increasing steadily.

Server bandwidth usage has risen considerably over the past two days.

What are you thoughts on maybe trying to get a new IP range allocated to the server?
I suspect this would work if the bot is simply pounding the IP address.
If this is a domain specific attack, then i suposed it will not be a solution.

If move the mail service to an edge device will that help?
Firewall all incomming connections on 25 and only allow from the edge box.
Force all clients to use port 587 for sending email.

Any suggestion and ideas will be most welcome.
Mark Brindley
2Large Networks - Web solutions that work
Post Reply