store | blogs | forums | twitter | facebook | wiki | downloads | support portal
Atomic Secure Linux
It is currently Sun Dec 15, 2019 2:58 pm

» Feed - Atomicorp

All times are UTC - 5 hours [ DST ]




Post new topic Reply to topic  [ 39 posts ]  Go to page Previous  1, 2, 3  Next
Author Message
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Mon Aug 05, 2013 1:42 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin

Joined: Wed Dec 31, 1969 8:00 pm
Posts: 8329
Location: earth
What do the packets look like? Just a SYN or a full 3-way handshake?


Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Mon Aug 05, 2013 2:06 pm 
Offline
Forum Regular
Forum Regular

Joined: Sat Dec 11, 2004 2:33 pm
Posts: 243
Location: South Africa
@faris

Thank you so much for taking the time to look.
I really appreciate all your time and effort.

I really hope somebody else here will be able to shed some light.
My clients are starting to rant and long standing clients are also threatening to leave :(

As a temp measure I have started moving important clients over to a new PSA 11.5 server.
Just hope the problem does not migrate as well.

_________________
Mark Brindley
2Large Networks - Web solutions that work


Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Mon Aug 05, 2013 4:32 pm 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Thu Dec 09, 2004 11:19 am
Posts: 2321
There's more than just a SYN, but...this is outside my knowledge to track

Here's a couple of screen shots of a fragment.


Attachments:
File comment: fragment 2
ws1.jpg
ws1.jpg [ 232.31 KiB | Viewed 7832 times ]
File comment: wireshark fragment
ws2a.jpg
ws2a.jpg [ 246 KiB | Viewed 7832 times ]

_________________
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Mon Aug 05, 2013 4:57 pm 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Thu Dec 09, 2004 11:19 am
Posts: 2321
Just to emphasise this, if I search for RCPT TO or FROM in one of the 20Mb captures (just port 25, remember), I get two or three hits, no more, and these seem to be part of a full conversation. The rest is all this "noise".

_________________
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>


Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Mon Aug 05, 2013 5:12 pm 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Thu Dec 09, 2004 11:19 am
Posts: 2321
Oh. I noticed I was sorting by IP in the screen shots I last sent.

Here's one ordered by time.


Attachments:
File comment: ws3
ws3a.jpg
ws3a.jpg [ 241.18 KiB | Viewed 7828 times ]

_________________
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Tue Aug 06, 2013 6:47 am 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Thu Dec 09, 2004 11:19 am
Posts: 2321
OK, OK, so what I've been posting has probably been incomprehensible. Blame it on a solid afternoon being deluged with data. I've totally simplified it and hopefully someone can help with this now.

GOOD.JPG
Attachment:
File comment: GOOD.JPG
good.jpg
good.jpg [ 53.26 KiB | Viewed 7825 times ]

Above is a screen shot of what I expect to see in terms of an SMTP conversation (from a spammer). It is an attempt to relay using the server in question. It follows the form I expect: server says 220, remote issues HELO, server says 250, remote gives RCTP TO, server says 250 and so on and so forth..

I note, incidentally, than in this type of situation there's no domain local to the server being mentioned, so we have no new data to work from.


BAD.JPG
Attachment:
File comment: BAD.JPG
bad.jpg
bad.jpg [ 119.31 KiB | Viewed 7825 times ]

Above is screen shot of an example of what happens with these "bad" connections that are driving is nuts.

** They all result in an "Unimplemented" response from the server. All of them. Every single one that I've checked.

You'll see now what I've talking about in terms of "noise". The third line down (31933) contains a big packet of...I don't know what. It isn't until you get three lines from the bottom before the server says 220 and then immediately follows it up with a 502 "Unimplemented".

This is what occurs in each of these thousands of connections that I've looked at.

Is this confirming what I thought? That the spambot is sending data without waiting (hence the earlytalker filter I mentioned before kicking in), so what we're seeing in line 31933 is actually somewhere in the middle of the email, maybe a fragment of its contents, with any reference to a RCPT TO or MAIL FROM long gone?

In these screen shots, I'm searching for conversations by IP address. So this is the "complete" capture of the conversation for this IP (though I note it says "[truncated]" for the command line, presumably because it is long).

And if so.....does this mean we're stuffed in terms of trying to figure out if there's a specific domain being targeted?

_________________
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>


Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Tue Aug 06, 2013 8:37 am 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin

Joined: Wed Dec 31, 1969 8:00 pm
Posts: 8329
Location: earth
To help make more sense of this, check out the decoder in wireshark, look under Analyze, Decode TCP stream. Select one of the sessions that dont make any sense and run that against it. It will reassemble the whole session in a more readable human output.

Im just speculating here, but that looks like its encrypted, they're probably trying to do a starttls and its not compatible with the MTA, so thats why you're seeing the 502 error.


Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Tue Aug 06, 2013 9:55 am 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Thu Dec 09, 2004 11:19 am
Posts: 2321
If I'm doing this correctly, they all look something like this: noise > server response > one or more unimplemented responses

e.g.

Code:
.v=d..D..@".^A.d.N.dk..e..;.@
"e.."eF........n..o..eo.6..J.e.. f.. fSp9f.T.....f.I....Of..4.#.g.3..f....0..f.k7g..Pg...gS.).y..g...gF.>..FX...8h...g.5hh*..h.}.h.g.h..<...<...o...3..1...C.jZ>.iu
{.1>{..p{...{...{...{...{...{...{.."|.."|(.<|(.<|5.U|O`.|v..|...|.^.}.^.}.^.}.*:}..l}.\.}.\.}.\.}1..}...}#..}...~,'8~9.Q~F.j~

220 server-domain.tld ESMTP

502 unimplemented (#5.5.1)

502 unimplemented (#5.5.1)

502 unimplemented (#5.5.1)

502 unimplemented (#5.5.1)

502 unimplemented (#5.5.1)


As far as I can tell, the server in question does support TLS (passthrough from spamdyke to qmail) and I specifically remember seeing a tls encrypted message being received and processed correctly when I was staring at the logs. I also tried doing tcpdump with tls deliberately disabled and they look the same so .... I don't know.

Here's a normal (refused spam) one:

Code:
220 server-hostname.tld ESMTP
EHLO [205.185.139.64]
250-server-hostname.tld
250-AUTH=LOGIN CRAM-MD5 PLAIN
250-AUTH LOGIN CRAM-MD5 PLAIN
250-STARTTLS
250-PIPELINING
250 8BITMIME
MAIL FROM:<name@some-other-domain.tld>
RCPT TO:<name@some-domain.tld>
DATA
250 ok
421 Refused. You have no reverse DNS entry.
421 Refused. You have no reverse DNS entry.
QUIT
221 Refused. You have no reverse DNS entry.

_________________
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>


Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Tue Aug 06, 2013 11:09 am 
Offline
Forum Regular
Forum Regular

Joined: Sat Mar 28, 2009 6:58 pm
Posts: 879
Location: Germany
I know thats not the solution and totally not good but it is a strange behaviour.
What happens if you disable TLS for a moment by setting this in spamdyke conf:
tls-level:none

Quote:
none: Do not provide or allow TLS, even if qmail supports it. qmail's attempt to advertise its TLS support will be hidden and the remote server's request for TLS will be denied.

http://www.spamdyke.org/documentation/README.html#TLS

of course this will cause some trouble for clients using tls but might be better than the ongoing flooding.
or at least it makes it possible to track it down a little bit.


Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Tue Aug 06, 2013 11:22 am 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin

Joined: Wed Dec 31, 1969 8:00 pm
Posts: 8329
Location: earth
The client certainly isnt doing any kind of error checking there, its just dumping everythng to MTA without waiting for it. On the plus side it means this douchebags spam client doesnt actually work, which could maybe be exploited as a way to test for spammer clients. Like greylisting, you throw back some kind of error that a smarter MUA would retry on.


Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Tue Aug 06, 2013 1:11 pm 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Thu Dec 09, 2004 11:19 am
Posts: 2321
I have tried disabling TLS completely and it makes no difference at all -- the traffic continues and the logs are the same.

So, basically broken spambot, then? But spamdyke can block them using a 2 second timeout. Even without it, qmail doesn't give them the time of day.
The thing is, the load is killing the server, even though it doesn't get past qmails greeting > unimplemented.

That's what I want to mitigate. I just don't have any ideas how. If it was directed at a particular domain, we would kill that domain off/move it. But we don't have any way to find out, because the spambot has got well past that stage before any data gets captures and qmail responds.

_________________
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>


Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Tue Aug 06, 2013 2:39 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin

Joined: Wed Dec 31, 1969 8:00 pm
Posts: 8329
Location: earth
Is there anything in the logs that tells you when its a spambot client? Where Im going with this is an rule that detects the client IP and sets a shun on it.


Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Tue Aug 06, 2013 4:02 pm 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Thu Dec 09, 2004 11:19 am
Posts: 2321
Yes and no. If I set spamdyke logging up one notch, there's a clear FILTER: EARLYTALKER (or similar - I forget the exact format), but the offending IP is not logged on that line.

I would need to see if Sam (Mr Spamdyke, if he'll forgive me for using that term) would help me patch the qmail code to add the offending IP to the log entry.

I am concerned, however, than even if we manage to do this, the sheer number of blocked IPs may result in 1000s of iptables entries. There's plenty of RAM in the system ...8Gb, 16Gb --Kram? I can't remember and I think it has a reasonable amount free to play with.

_________________
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>


Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Tue Aug 06, 2013 4:46 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin

Joined: Wed Dec 31, 1969 8:00 pm
Posts: 8329
Location: earth
Ive definitely got a solution for large ip lists (called ipsets), in testing Ive done millions of IP's in a single rule. No impact on either memory or performance (looks prettier too!). Its part of the xtables-addon package in the ASL kernels, for non-ASL kernels we could probably do a few hundred thousand the old way without much impact.


Top
 Profile  
Reply with quote  
 Post subject: Re: Extreme incoming Email killing CPu
Unread postPosted: Tue Aug 06, 2013 4:57 pm 
Offline
Forum Regular
Forum Regular

Joined: Sat Dec 11, 2004 2:33 pm
Posts: 243
Location: South Africa
Hi Faris,

Here is the output from free

Code:
total       used       free     shared    buffers     cached
Mem:      16074312   15630168     444144          0     485564    5633296
-/+ buffers/cache:    9511308    6563004
Swap:      4198972     746540    3452432


Mail delivery is totally out of whack now, mail is coming in 5 hours late.
Connections to the server seem to be increasing steadily.

Server bandwidth usage has risen considerably over the past two days.

What are you thoughts on maybe trying to get a new IP range allocated to the server?
I suspect this would work if the bot is simply pounding the IP address.
If this is a domain specific attack, then i suposed it will not be a solution.

If move the mail service to an edge device will that help?
Firewall all incomming connections on 25 and only allow from the edge box.
Force all clients to use port 587 for sending email.

Any suggestion and ideas will be most welcome.

_________________
Mark Brindley
2Large Networks - Web solutions that work


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 39 posts ]  Go to page Previous  1, 2, 3  Next

» Feed - Atomicorp

All times are UTC - 5 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group