Extreme incoming Email killing CPu
-
- Atomicorp Staff - Site Admin
- Posts: 8355
- Joined: Wed Dec 31, 1969 8:00 pm
- Location: earth
- Contact:
Re: Extreme incoming Email killing CPu
What do the packets look like? Just a SYN or a full 3-way handshake?
Re: Extreme incoming Email killing CPu
@faris
Thank you so much for taking the time to look.
I really appreciate all your time and effort.
I really hope somebody else here will be able to shed some light.
My clients are starting to rant and long standing clients are also threatening to leave
As a temp measure I have started moving important clients over to a new PSA 11.5 server.
Just hope the problem does not migrate as well.
Thank you so much for taking the time to look.
I really appreciate all your time and effort.
I really hope somebody else here will be able to shed some light.
My clients are starting to rant and long standing clients are also threatening to leave
As a temp measure I have started moving important clients over to a new PSA 11.5 server.
Just hope the problem does not migrate as well.
Mark Brindley
2Large Networks - Web solutions that work
2Large Networks - Web solutions that work
Re: Extreme incoming Email killing CPu
There's more than just a SYN, but...this is outside my knowledge to track
Here's a couple of screen shots of a fragment.
Here's a couple of screen shots of a fragment.
- Attachments
-
- fragment 2
- ws1.jpg (232.31 KiB) Viewed 15476 times
-
- wireshark fragment
- ws2a.jpg (246 KiB) Viewed 15476 times
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
Re: Extreme incoming Email killing CPu
Just to emphasise this, if I search for RCPT TO or FROM in one of the 20Mb captures (just port 25, remember), I get two or three hits, no more, and these seem to be part of a full conversation. The rest is all this "noise".
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
Re: Extreme incoming Email killing CPu
Oh. I noticed I was sorting by IP in the screen shots I last sent.
Here's one ordered by time.
Here's one ordered by time.
- Attachments
-
- ws3
- ws3a.jpg (241.18 KiB) Viewed 15472 times
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
Re: Extreme incoming Email killing CPu
OK, OK, so what I've been posting has probably been incomprehensible. Blame it on a solid afternoon being deluged with data. I've totally simplified it and hopefully someone can help with this now.
GOOD.JPG Above is a screen shot of what I expect to see in terms of an SMTP conversation (from a spammer). It is an attempt to relay using the server in question. It follows the form I expect: server says 220, remote issues HELO, server says 250, remote gives RCTP TO, server says 250 and so on and so forth..
I note, incidentally, than in this type of situation there's no domain local to the server being mentioned, so we have no new data to work from.
BAD.JPG Above is screen shot of an example of what happens with these "bad" connections that are driving is nuts.
** They all result in an "Unimplemented" response from the server. All of them. Every single one that I've checked.
You'll see now what I've talking about in terms of "noise". The third line down (31933) contains a big packet of...I don't know what. It isn't until you get three lines from the bottom before the server says 220 and then immediately follows it up with a 502 "Unimplemented".
This is what occurs in each of these thousands of connections that I've looked at.
Is this confirming what I thought? That the spambot is sending data without waiting (hence the earlytalker filter I mentioned before kicking in), so what we're seeing in line 31933 is actually somewhere in the middle of the email, maybe a fragment of its contents, with any reference to a RCPT TO or MAIL FROM long gone?
In these screen shots, I'm searching for conversations by IP address. So this is the "complete" capture of the conversation for this IP (though I note it says "[truncated]" for the command line, presumably because it is long).
And if so.....does this mean we're stuffed in terms of trying to figure out if there's a specific domain being targeted?
GOOD.JPG Above is a screen shot of what I expect to see in terms of an SMTP conversation (from a spammer). It is an attempt to relay using the server in question. It follows the form I expect: server says 220, remote issues HELO, server says 250, remote gives RCTP TO, server says 250 and so on and so forth..
I note, incidentally, than in this type of situation there's no domain local to the server being mentioned, so we have no new data to work from.
BAD.JPG Above is screen shot of an example of what happens with these "bad" connections that are driving is nuts.
** They all result in an "Unimplemented" response from the server. All of them. Every single one that I've checked.
You'll see now what I've talking about in terms of "noise". The third line down (31933) contains a big packet of...I don't know what. It isn't until you get three lines from the bottom before the server says 220 and then immediately follows it up with a 502 "Unimplemented".
This is what occurs in each of these thousands of connections that I've looked at.
Is this confirming what I thought? That the spambot is sending data without waiting (hence the earlytalker filter I mentioned before kicking in), so what we're seeing in line 31933 is actually somewhere in the middle of the email, maybe a fragment of its contents, with any reference to a RCPT TO or MAIL FROM long gone?
In these screen shots, I'm searching for conversations by IP address. So this is the "complete" capture of the conversation for this IP (though I note it says "[truncated]" for the command line, presumably because it is long).
And if so.....does this mean we're stuffed in terms of trying to figure out if there's a specific domain being targeted?
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
-
- Atomicorp Staff - Site Admin
- Posts: 8355
- Joined: Wed Dec 31, 1969 8:00 pm
- Location: earth
- Contact:
Re: Extreme incoming Email killing CPu
To help make more sense of this, check out the decoder in wireshark, look under Analyze, Decode TCP stream. Select one of the sessions that dont make any sense and run that against it. It will reassemble the whole session in a more readable human output.
Im just speculating here, but that looks like its encrypted, they're probably trying to do a starttls and its not compatible with the MTA, so thats why you're seeing the 502 error.
Im just speculating here, but that looks like its encrypted, they're probably trying to do a starttls and its not compatible with the MTA, so thats why you're seeing the 502 error.
Re: Extreme incoming Email killing CPu
If I'm doing this correctly, they all look something like this: noise > server response > one or more unimplemented responses
e.g.
As far as I can tell, the server in question does support TLS (passthrough from spamdyke to qmail) and I specifically remember seeing a tls encrypted message being received and processed correctly when I was staring at the logs. I also tried doing tcpdump with tls deliberately disabled and they look the same so .... I don't know.
Here's a normal (refused spam) one:
e.g.
Code: Select all
.v=d..D..@".^A.d.N.dk..e..;.@
"e.."eF........n..o..eo.6..J.e.. f.. fSp9f.T.....f.I....Of..4.#.g.3..f....0..f.k7g..Pg...gS.).y..g...gF.>..FX...8h...g.5hh*..h.}.h.g.h..<...<...o...3..1...C.jZ>.iu
{.1>{..p{...{...{...{...{...{...{.."|.."|(.<|(.<|5.U|O`.|v..|...|.^.}.^.}.^.}.*:}..l}.\.}.\.}.\.}1..}...}#..}...~,'8~9.Q~F.j~
220 server-domain.tld ESMTP
502 unimplemented (#5.5.1)
502 unimplemented (#5.5.1)
502 unimplemented (#5.5.1)
502 unimplemented (#5.5.1)
502 unimplemented (#5.5.1)
Here's a normal (refused spam) one:
Code: Select all
220 server-hostname.tld ESMTP
EHLO [205.185.139.64]
250-server-hostname.tld
250-AUTH=LOGIN CRAM-MD5 PLAIN
250-AUTH LOGIN CRAM-MD5 PLAIN
250-STARTTLS
250-PIPELINING
250 8BITMIME
MAIL FROM:<name@some-other-domain.tld>
RCPT TO:<name@some-domain.tld>
DATA
250 ok
421 Refused. You have no reverse DNS entry.
421 Refused. You have no reverse DNS entry.
QUIT
221 Refused. You have no reverse DNS entry.
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
Re: Extreme incoming Email killing CPu
I know thats not the solution and totally not good but it is a strange behaviour.
What happens if you disable TLS for a moment by setting this in spamdyke conf:
tls-level:none
of course this will cause some trouble for clients using tls but might be better than the ongoing flooding.
or at least it makes it possible to track it down a little bit.
What happens if you disable TLS for a moment by setting this in spamdyke conf:
tls-level:none
http://www.spamdyke.org/documentation/README.html#TLSnone: Do not provide or allow TLS, even if qmail supports it. qmail's attempt to advertise its TLS support will be hidden and the remote server's request for TLS will be denied.
of course this will cause some trouble for clients using tls but might be better than the ongoing flooding.
or at least it makes it possible to track it down a little bit.
-
- Atomicorp Staff - Site Admin
- Posts: 8355
- Joined: Wed Dec 31, 1969 8:00 pm
- Location: earth
- Contact:
Re: Extreme incoming Email killing CPu
The client certainly isnt doing any kind of error checking there, its just dumping everythng to MTA without waiting for it. On the plus side it means this douchebags spam client doesnt actually work, which could maybe be exploited as a way to test for spammer clients. Like greylisting, you throw back some kind of error that a smarter MUA would retry on.
Re: Extreme incoming Email killing CPu
I have tried disabling TLS completely and it makes no difference at all -- the traffic continues and the logs are the same.
So, basically broken spambot, then? But spamdyke can block them using a 2 second timeout. Even without it, qmail doesn't give them the time of day.
The thing is, the load is killing the server, even though it doesn't get past qmails greeting > unimplemented.
That's what I want to mitigate. I just don't have any ideas how. If it was directed at a particular domain, we would kill that domain off/move it. But we don't have any way to find out, because the spambot has got well past that stage before any data gets captures and qmail responds.
So, basically broken spambot, then? But spamdyke can block them using a 2 second timeout. Even without it, qmail doesn't give them the time of day.
The thing is, the load is killing the server, even though it doesn't get past qmails greeting > unimplemented.
That's what I want to mitigate. I just don't have any ideas how. If it was directed at a particular domain, we would kill that domain off/move it. But we don't have any way to find out, because the spambot has got well past that stage before any data gets captures and qmail responds.
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
-
- Atomicorp Staff - Site Admin
- Posts: 8355
- Joined: Wed Dec 31, 1969 8:00 pm
- Location: earth
- Contact:
Re: Extreme incoming Email killing CPu
Is there anything in the logs that tells you when its a spambot client? Where Im going with this is an rule that detects the client IP and sets a shun on it.
Re: Extreme incoming Email killing CPu
Yes and no. If I set spamdyke logging up one notch, there's a clear FILTER: EARLYTALKER (or similar - I forget the exact format), but the offending IP is not logged on that line.
I would need to see if Sam (Mr Spamdyke, if he'll forgive me for using that term) would help me patch the qmail code to add the offending IP to the log entry.
I am concerned, however, than even if we manage to do this, the sheer number of blocked IPs may result in 1000s of iptables entries. There's plenty of RAM in the system ...8Gb, 16Gb --Kram? I can't remember and I think it has a reasonable amount free to play with.
I would need to see if Sam (Mr Spamdyke, if he'll forgive me for using that term) would help me patch the qmail code to add the offending IP to the log entry.
I am concerned, however, than even if we manage to do this, the sheer number of blocked IPs may result in 1000s of iptables entries. There's plenty of RAM in the system ...8Gb, 16Gb --Kram? I can't remember and I think it has a reasonable amount free to play with.
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
-
- Atomicorp Staff - Site Admin
- Posts: 8355
- Joined: Wed Dec 31, 1969 8:00 pm
- Location: earth
- Contact:
Re: Extreme incoming Email killing CPu
Ive definitely got a solution for large ip lists (called ipsets), in testing Ive done millions of IP's in a single rule. No impact on either memory or performance (looks prettier too!). Its part of the xtables-addon package in the ASL kernels, for non-ASL kernels we could probably do a few hundred thousand the old way without much impact.
Re: Extreme incoming Email killing CPu
Hi Faris,
Here is the output from free
Mail delivery is totally out of whack now, mail is coming in 5 hours late.
Connections to the server seem to be increasing steadily.
Server bandwidth usage has risen considerably over the past two days.
What are you thoughts on maybe trying to get a new IP range allocated to the server?
I suspect this would work if the bot is simply pounding the IP address.
If this is a domain specific attack, then i suposed it will not be a solution.
If move the mail service to an edge device will that help?
Firewall all incomming connections on 25 and only allow from the edge box.
Force all clients to use port 587 for sending email.
Any suggestion and ideas will be most welcome.
Here is the output from free
Code: Select all
total used free shared buffers cached
Mem: 16074312 15630168 444144 0 485564 5633296
-/+ buffers/cache: 9511308 6563004
Swap: 4198972 746540 3452432
Connections to the server seem to be increasing steadily.
Server bandwidth usage has risen considerably over the past two days.
What are you thoughts on maybe trying to get a new IP range allocated to the server?
I suspect this would work if the bot is simply pounding the IP address.
If this is a domain specific attack, then i suposed it will not be a solution.
If move the mail service to an edge device will that help?
Firewall all incomming connections on 25 and only allow from the edge box.
Force all clients to use port 587 for sending email.
Any suggestion and ideas will be most welcome.
Mark Brindley
2Large Networks - Web solutions that work
2Large Networks - Web solutions that work