Flooded with Spam

Forum for getting help with Project Gamera, Spamassassin, Clamav, qmail-scanner and other anti-spam tools.
netcomseth
Forum User
Forum User
Posts: 10
Joined: Mon Feb 18, 2008 6:00 pm

Flooded with Spam

Unread post by netcomseth »

Hello all,

I have a hosted server running RHELS 4 and Plesk 8.2. We host mail for a number of domains, and suddenly our mail queue is filling with thousands of Failure Notice and SPAM emails each hour.

I have used qmHandle to examine headers to try and find out which uid is sending the mails, however all of the 'Failure Notice" say invoked for bounce, and the spams say invoked from network.

Where should I look next to continue to track down the issue?

Thanks,
Seth
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8355
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Unread post by scott »

Ive got some rough procedures here:

http://www.atomicorp.com/wiki/index.php/Spam
netcomseth
Forum User
Forum User
Posts: 10
Joined: Mon Feb 18, 2008 6:00 pm

Unread post by netcomseth »

Thanks for the Reply Scott! I did come across that page earlier, and that got me to where I am. the problem is as I mentioned above, there are no uid's listed, just invoked for bounce and invoked from network. This is why I'm lost, if I had a uid I feel I would be close to solving the mystery.

What would the next step be? I am including an example of each message:

Failure Notice::
--------------
MESSAGE NUMBER 14012046
--------------
Received: (qmail 1606 invoked for bounce); 18 Feb 2008 15:31:10 -0800
Date: 18 Feb 2008 15:31:10 -0800
From: MAILER-DAEMON@"mydomain"
To: haskel@talk21.com
Subject: failure notice

Hi. This is the qmail-send program at mx.mydomain.com.
I'm afraid I wasn't able to deliver your message to the following addresses.
This is a permanent error; I've given up. Sorry it didn't work out.

<imsrepublic@mydomain.com>:
Sorry. Although I'm listed as a best-preference MX or A for that host,
it isn't in my control/locals file, so I don't treat it as local. (#5.4.6)

SPAM::

--------------
MESSAGE NUMBER 14010263
--------------
Received: (qmail 5011 invoked from network); 18 Feb 2008 15:10:56 -0800
Received: from dsl.dynamic8121477203.ttnet.net.tr (81.214.77.203)
by mydomain.com with SMTP; 18 Feb 2008 15:10:55 -0800
Received-SPF: none (mydomain.com: domain at atomic.com does not designate permitted sender hosts)
Message-ID: <000501c87283$012356f8$9913e0ad@mwwux>
From: "Omega Watches" <rafael@atomic.com>
To: "Replica Watches" <qcold@mansfieldent.com>
Subject: Just waiting for a Breitling
Date: Mon, 18 Feb 2008 21:23:33 +0000
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="----=_NextPart_000_0002_01C87283.0122176B"
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2900.3138
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3198

This is a multi-part message in MIME format.
_____________________________________________

Where would you go next?
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8355
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Unread post by scott »

This part here tells you what you need to know:

Received: (qmail 5011 invoked from network); 18 Feb 2008 15:10:56 -0800
Received: from dsl.dynamic8121477203.ttnet.net.tr (81.214.77.203)

The message is coming over the network from that host, either because the IP is whitelisted (ie, poplocking) or they're using a compromised smtp_auth account.
netcomseth
Forum User
Forum User
Posts: 10
Joined: Mon Feb 18, 2008 6:00 pm

Unread post by netcomseth »

Thanks for the reply.

I have been looking through the logs to try and find out which account may be compromised. But is seems that qmail doesn't log which user is sending mail? Is this correct? Is there any way to turn up the logging to show this?

As far as white listing, all I have in my plesk white list is 127.0.0.1/32 as I have seen recommended in multiple forums. Could spammers still be using my server with this setting?

Thanks,
Seth
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8355
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Unread post by scott »

Its an smtp_auth account, unless you allow poplocking and someone is abusing that. It should be logging those smtp_auth logins to the /usr/local/psa/var/log/maillog file.
netcomseth
Forum User
Forum User
Posts: 10
Joined: Mon Feb 18, 2008 6:00 pm

Unread post by netcomseth »

I'm not sure I exactly understand the idea of poplocking. I will look into it while I await the next reply.

Here is a sample of my /usr/local/psa/var/log/maillog
Feb 19 11:06:18 netcomwest qmail-queue-handlers[2660]: starter: submitter[2662] exited normally
Feb 19 11:06:18 netcomwest qmail: 1203447978.757627 starting delivery 25466: msg 14010033 to remote postmaster@mx.netcomwest.com
Feb 19 11:06:18 netcomwest qmail: 1203447978.757672 status: local 0/30 remote 20/20
Feb 19 11:06:18 netcomwest qmail: 1203447978.757702 new msg 14009770
Feb 19 11:06:18 netcomwest qmail: 1203447978.757732 info msg 14009770: bytes 13976 from <dapper10@earthlink.net> qp 2662 uid 2020
Feb 19 11:06:18 netcomwest qmail-remote-handlers[2663]: Handlers Filter before-remote for qmail started ...
Feb 19 11:06:18 netcomwest qmail: 1203447978.764977 new msg 14009771
Feb 19 11:06:18 netcomwest qmail: 1203447978.765021 info msg 14009771: bytes 3443 from <#@[]> qp 2657 uid 2522
Feb 19 11:06:18 netcomwest qmail-remote-handlers[2663]: from=#@[]
Feb 19 11:06:18 netcomwest qmail-remote-handlers[2663]: to=postmaster@mx.netcomwest.com
Feb 19 11:06:18 netcomwest qmail: 1203447978.785437 delivery 25466: failure: Sorry._Although_I'm_listed_as_a_best-preference_MX_or_A_for_that_host,/it_isn't_in_my_control/locals_file,_so_I_don't_treat_it_as_local._(#5.4.6)/
Feb 19 11:06:18 netcomwest qmail: 1203447978.785499 status: local 0/30 remote 19/20
Feb 19 11:06:18 netcomwest qmail: 1203447978.785529 triple bounce: discarding bounce/14010033
Feb 19 11:06:18 netcomwest qmail: 1203447978.785558 end msg 14010033
Feb 19 11:06:18 netcomwest qmail: 1203447978.807934 starting delivery 25467: msg 14010008 to remote postmaster@mx.netcomwest.com
Feb 19 11:06:18 netcomwest qmail: 1203447978.807976 status: local 0/30 remote 20/20
Feb 19 11:06:18 netcomwest qmail-remote-handlers[2667]: Handlers Filter before-remote for qmail started ...
Feb 19 11:06:18 netcomwest qmail-remote-handlers[2667]: from=#@[]
Feb 19 11:06:18 netcomwest qmail-remote-handlers[2667]: to=postmaster@mx.netcomwest.com
Feb 19 11:06:18 netcomwest qmail: 1203447978.829331 delivery 25467: failure: Sorry._Although_I'm_listed_as_a_best-preference_MX_or_A_for_that_host,/it_isn't_in_my_control/locals_file,_so_I_don't_treat_it_as_local._(#5.4.6)/
Feb 19 11:06:18 netcomwest qmail: 1203447978.829396 status: local 0/30 remote 19/20
Feb 19 11:06:18 netcomwest qmail: 1203447978.829426 triple bounce: discarding bounce/14010008
Feb 19 11:06:18 netcomwest qmail: 1203447978.829455 end msg 14010008
Feb 19 11:06:18 netcomwest relaylock: /var/qmail/bin/relaylock: mail from 124.122.204.111:51145 (ppp-124-122-204-111.revip2.asianet.co.th)
Feb 19 11:06:18 netcomwest qmail: 1203447978.854096 starting delivery 25468: msg 14009946 to remote krgturner@mansfieldent.com
Feb 19 11:06:18 netcomwest qmail: 1203447978.854142 status: local 0/30 remote 20/20
Feb 19 11:06:18 netcomwest qmail-remote-handlers[2672]: Handlers Filter before-remote for qmail started ...
Feb 19 11:06:18 netcomwest qmail-remote-handlers[2672]: from=cuyxs@adelphia.com
Feb 19 11:06:18 netcomwest qmail-remote-handlers[2672]: to=krgturner@mansfieldent.com
Feb 19 11:06:18 netcomwest qmail: 1203447978.894065 delivery 25468: failure: Sorry._Although_I'm_listed_as_a_best-preference_MX_or_A_for_that_host,/it_isn't_in_my_control/locals_file,_so_I_don't_treat_it_as_local._(#5.4.6)/
Feb 19 11:06:18 netcomwest qmail: 1203447978.894129 status: local 0/30 remote 19/20
Feb 19 11:06:18 netcomwest qmail-queue[2674]: mail: all addreses are uncheckable - need to skip scanning (by deny mode)
Feb 19 11:06:18 netcomwest qmail-queue[2674]: scan: the message(drweb.tmp.DOr8Vp) sent by to cuyxs@adelphia.com should be passed without checks, because contains uncheckable addresses
Feb 19 11:06:18 netcomwest qmail-queue-handlers[2675]: Handlers Filter before-queue for qmail started ...
Feb 19 11:06:18 netcomwest qmail-queue-handlers[2675]: from=
Feb 19 11:06:18 netcomwest qmail-queue-handlers[2675]: to=cuyxs@adelphia.com
Feb 19 11:06:18 netcomwest qmail-queue-handlers[2675]: hook_dir = '/var/qmail//handlers/before-queue'
Feb 19 11:06:18 netcomwest qmail-queue-handlers[2675]: recipient[3] = 'cuyxs@adelphia.com'
Feb 19 11:06:18 netcomwest qmail-queue-handlers[2675]: handlers dir = '/var/qmail//handlers/before-queue/recipient/cuyxs@adelphia.com'
Feb 19 11:06:18 netcomwest qmail-queue-handlers[2675]: starter: submitter[2676] exited normally
Feb 19 11:06:18 netcomwest qmail: 1203447978.912998 bounce msg 14009946 qp 2674
Feb 19 11:06:18 netcomwest qmail: 1203447978.913043 end msg 14009946
Feb 19 11:06:18 netcomwest qmail: 1203447978.948248 starting delivery 25469: msg 14010013 to remote ig@mansfieldent.com
Feb 19 11:06:18 netcomwest qmail: 1203447978.949438 status: local 0/30 remote 20/20
Feb 19 11:06:18 netcomwest qmail: 1203447978.949473 new msg 14009775
Feb 19 11:06:18 netcomwest qmail: 1203447978.949503 info msg 14009775: bytes 14612 from <> qp 2676 uid 2522
Feb 19 11:06:18 netcomwest qmail-remote-handlers[2677]: Handlers Filter before-remote for qmail started ...
Feb 19 11:06:18 netcomwest qmail-remote-handlers[2677]: from=tequilaman909@pataskala.com
Feb 19 11:06:18 netcomwest qmail-remote-handlers[2677]: to=ig@mansfieldent.com
Feb 19 11:06:18 netcomwest qmail: 1203447978.968209 delivery 25469: failure: Sorry._Although_I'm_listed_as_a_best-preference_MX_or_A_for_that_host,/it_isn't_in_my_control/locals_file,_so_I_don't_treat_it_as_local._(#5.4.6)/
Feb 19 11:06:18 netcomwest qmail: 1203447978.968276 status: local 0/30 remote 19/20
Any futher ideas? the uid 2522 is user qmails

I feel like I am so close to figuring this out, but just out of reach... Thank you again for any help you can give me.
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8355
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Unread post by scott »

if its on, poplocking will whitelist the IP that the user is coming from when they authenticate over pop. This can be a problem if they come from a proxy or something.

You want to grep for smtp_auth, assuming its even being logged. Sometimes it likes to hide in /var/log/messages too.
netcomseth
Forum User
Forum User
Posts: 10
Joined: Mon Feb 18, 2008 6:00 pm

Unread post by netcomseth »

Thank you! That showed me what user had been logging in at all hours of day and night even though they had an empty mailbox. I have notified the user that I have changed their password.

I'll continue to monitor my logs and reply back with results. Thank you again Scott.

-Seth-
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8355
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Unread post by scott »

No problem, glad to see you found the source of the spam!
netcomseth
Forum User
Forum User
Posts: 10
Joined: Mon Feb 18, 2008 6:00 pm

Unread post by netcomseth »

This morning, queue was back up to 25,000... Tried to check the logs, but nothing came up...

This is out of hand, is there any way to stop this?
netcomseth
Forum User
Forum User
Posts: 10
Joined: Mon Feb 18, 2008 6:00 pm

Unread post by netcomseth »

Okay appears that my logs rotated last night...

Old or New Logs don't show anyone logging in repeatedly over the course of the night...

Yet, spams are still flowing in like crazy.

I have confirmed in PLESK that poplock is disabled. I have smtp authorization on. I have set reject to all domains for mail to non-existent users.

Where do I go from here? Is there another log I can check for a clue? Any settings I should check that could be leaving an open door? Will posting all my logs help?

Thanks in advance,
Seth
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8355
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Unread post by scott »

Same process, you need to find a spam message to identify the source. You could have multiple sources on one system (smtp_auth, and web apps)
netcomseth
Forum User
Forum User
Posts: 10
Joined: Mon Feb 18, 2008 6:00 pm

Unread post by netcomseth »

Hi scott,

I have gone through the same process, however maybe I'm not smart enough to figure this out. I can see ip's that spams are coming from (they seem to be all over the place.) The from's on the mails are all spoofed. I am not seeing any pattern as far as smtp_auth in logs pointing to a specific user(s).

I had run through a few checks that I found in other forums to look for rouge scripts and such. Can anyone suggest another way to tell if this is a web app, and not a user?

I appreciate any helpful responses,
Seth
CrK01
Forum User
Forum User
Posts: 94
Joined: Wed Jun 06, 2007 10:49 am

Unread post by CrK01 »

Hello,

I have similar problem few days ago, and it seems that ( in my case ) it was an "open relay" on my qmail server.

To solve this issue, ensure that Local Domains ( locals ) refeers only to localhost for example, and Accepted Domains ( rcpthosts ), ONLY domains listed is checked, and in this list, you must put all your domains.

I did this with webmin , in the qmail control section, but you can do it changing appropiate files.

In my case, the problem was solved.

Another possibility is that you have a malicious script in /tmp or /var/tmp sending spam. Ensure /tmp is free of this ( ls -la /tmp ) and there aren't any suspicious scripts ( perl scripts, etc )
Post Reply