Unexplainable mail not getting through

[coolemail]

I have a Linux server with Centos 5, Plesk 8.6, ASL, spamdyke (those are the main bits!). There are some emails not getting through for some reason, and I cannot fathom out why.

I sent a test email which did not get through to the Inbox

Apr 27 10:39:04 plesk2 qmail: 1240825144.031395 new msg 14814299
Apr 27 10:39:04 plesk2 qmail: 1240825144.031509 info msg 14814299: bytes 1424 from <test_email@hotmail.com> qp 1126 uid 2020
Apr 27 10:39:04 plesk2 qmail: 1240825144.045506 starting delivery 6678: msg 14814299 to local 47-info@domain1.com
Apr 27 10:39:04 plesk2 qmail: 1240825144.045576 status: local 1/10 remote 0/20
Apr 27 10:39:04 plesk2 qmail-local-handlers[1127]: Handlers Filter before-local for qmail started ...
Apr 27 10:39:04 plesk2 qmail-local-handlers[1127]: from=test_email@hotmail.com
Apr 27 10:39:04 plesk2 qmail-local-handlers[1127]: to=info@domain1.com
Apr 27 10:39:04 plesk2 spamd[2166]: spamd: got connection over /tmp/spamd_full.sock
Apr 27 10:39:04 plesk2 spamd[2166]: spamd: using default config for info@domain1.com: /var/qmail/mailnames/domain1.com/info/.spamassassin/user_prefs
Apr 27 10:39:04 plesk2 spamd[2166]: spamd: processing message <COL105-W24181160D28A7E47830F3EB4710@phx.gbl> for info@domain1.com:110
Apr 27 10:39:05 plesk2 spamd[2166]: spamd: identified spam (4.1/4.0) for info@expat-email.com:110 in 1.3 seconds, 1424 bytes.
Apr 27 10:39:05 plesk2 spamd[2166]: spamd: result: Y 4 - AWL,BAYES_99,HTML_MESSAGE,MIME_HTML_MOSTLY scantime=1.3,size=1424,user=info@domain1.com,uid=110,required_score=4.0,rhost=localhost,raddr=127.0.0.1,rport=/tmp/spamd_full.sock,mid=<COL105-W24181160D28A7E47830F3EB4710@phx.gbl>,bayes=1.000000,autolearn=no
Apr 27 10:39:05 plesk2 qmail: 1240825145.411898 delivery 6678: success: did_0+0+1/
Apr 27 10:39:05 plesk2 qmail: 1240825145.411953 status: local 0/10 remote 0/20
Apr 27 10:39:05 plesk2 qmail: 1240825145.411973 end msg 14814299

The following one DID get through from the same test email to another email address on domain1.com

Apr 27 10:53:35 plesk2 spamdyke[3014]: ALLOWED from: test_email@hotmail.com to: email2@domain1.com origin_ip: 65.55.34.143 origin_rdns: col0-omc3-s5.col0.hotmail.com auth: (unknown)
Apr 27 10:53:36 plesk2 qmail-queue[3026]: scan: the message(drweb.tmp.W7Skqk) sent by test_email@hotmail.com to email2@domain1.com is passed
Apr 27 10:53:36 plesk2 qmail-queue-handlers[3027]: Handlers Filter before-queue for qmail started ...
Apr 27 10:53:36 plesk2 qmail-queue-handlers[3027]: from=test_email@hotmail.com
Apr 27 10:53:36 plesk2 qmail-queue-handlers[3027]: to=email2@domain1.com
Apr 27 10:53:36 plesk2 qmail-queue-handlers[3027]: hook_dir = '/var/qmail//handlers/before-queue'
Apr 27 10:53:36 plesk2 qmail-queue-handlers[3027]: recipient[3] = 'email2@domain1.com'
Apr 27 10:53:36 plesk2 qmail-queue-handlers[3027]: handlers dir = '/var/qmail//handlers/before-queue/recipient/email2@domain1.com'
Apr 27 10:53:36 plesk2 qmail: 1240826016.297787 new msg 14814299
Apr 27 10:53:36 plesk2 qmail: 1240826016.297905 info msg 14814299: bytes 1447 from <test_email@hotmail.com> qp 3028 uid 2020
Apr 27 10:53:36 plesk2 qmail-queue-handlers[3027]: starter: submitter[3028] exited normally
Apr 27 10:53:36 plesk2 qmail: 1240826016.314443 starting delivery 6717: msg 14814299 to local 47-email2@domain1.com
Apr 27 10:53:36 plesk2 qmail: 1240826016.314579 status: local 1/10 remote 0/20
Apr 27 10:53:36 plesk2 qmail-local-handlers[3029]: Handlers Filter before-local for qmail started ...
Apr 27 10:53:36 plesk2 qmail-local-handlers[3029]: from=test_email@hotmail.com
Apr 27 10:53:36 plesk2 qmail-local-handlers[3029]: to=email2@domain1.com
Apr 27 10:53:36 plesk2 spamd[2166]: spamd: got connection over /tmp/spamd_full.sock
Apr 27 10:53:36 plesk2 spamd[2166]: spamd: using default config for email2@domain1.com: /var/qmail/mailnames/domain1.com/email2/.spamassassin/user_prefs
Apr 27 10:53:36 plesk2 spamd[2166]: spamd: processing message <COL105-W652FB36A4A385D92B721F7B4710@phx.gbl> for email2@domain1.com:110
Apr 27 10:53:37 plesk2 spamd[2166]: spamd: clean message (-2.1/4.0) for email2@domain1.com:110 in 1.6 seconds, 1447 bytes.
Apr 27 10:53:37 plesk2 spamd[2166]: spamd: result: . -2 - AWL,BAYES_00,HTML_MESSAGE,MIME_HTML_MOSTLY scantime=1.6,size=1447,user=email2@domain1.com,uid=110,required_score=4.0,rhost=localhost,raddr=127.0.0.1,rport=/tmp/spamd_full.sock,mid=<COL105-W652FB36A4A385D92B721F7B4710@phx.gbl>,bayes=0.000000,autolearn=ham
Apr 27 10:53:37 plesk2 qmail-local-handlers[3047]: Handlers Filter before-local for qmail started ...
Apr 27 10:53:37 plesk2 qmail-local-handlers[3047]: from=test_email@hotmail.com
Apr 27 10:53:37 plesk2 qmail-local-handlers[3047]: to=email2@domain1.com
Apr 27 10:53:37 plesk2 qmail: 1240826017.914094 delivery 6717: success: did_0+0+3/did_0+0+1/
Apr 27 10:53:37 plesk2 qmail: 1240826017.914144 status: local 0/10 remote 0/20
Apr 27 10:53:37 plesk2 qmail: 1240826017.914163 end msg 14814299

Can anyone help me shed light on what is causing that failure. No failure notice was sent to test_email@hotmail.com. It seems strange that this is happening with email to a single email address on one domain (that we can pinpoint so far) because I sent the email to 2 different email addresses on domain1.com which got through, and only the one known one failed. I first noticed this because a known email from mailman requesting me to authorise an email did not get delivered.

Other emails from external domains to info@domain1.com are getting delivered, so I have no idea of the scale of what may be getting rejected.

The most recent major change to the server was to remove qgreylist and to install spamdyke, if that gives anything that will help.

Many thanks in advance, and I do hope that someone can help here.

[faris]

Well, there's one obvious difference, and that's that the first message was identified as spam, while the second one wasn't.

Have a look in /var/log/messages to see if there is any indication that something else happened to the email.

Also have a look in /var/qmail/mailnames/[domain.com]/[email-address]/Maildir/ (I think --- I'm making it up as I go along from memory again) and have a poke around the directories there to make sure it hasn't been moved to an imap folder by some anti-spam application or something?

I'm grasping at straws here. Maybe someone will spot something more sensible from the log you posted.

Faris.

[biggles]

Agree with faris. I must have something to do with the first message being marked as spam and the second isn't.

[coolemail]

Thank You for the 2 responses. I cannot see why the first email was marked as Spam, and the second was not (and I sent two emails to info, both of which were rejected). They only contained a subject line saying test. Is it possible to see why spamdyke rejected them, and which bit of the Spam checking gave it the high score that made it get rejected?

I have looked in /var/qmail/mailnames/[domain.com]/[email-address]/Maildir/ and the email is not there. It has not been put in any of the other folders by accident.

Is there a way to look for a particular bit of /var/log/messages because that file is huge. Can I drill down in it to get to the relevant time/message?

The other one that I am unable to track down is the server-issued automated email from mailman saying that an email needs authentication. That did not get through either, and if there were genuine emails requiring modification, I would have no idea which is not great.

Are there any files in /var/qmail/spamdyke that I can play with to allow them through? For example, can I add certain addresses to whitelist_senders which are then whitelisted on the checking? For exapmple, could I put in mailman-bounces@*.* in the whitelist_senders?

[biggles]

"grep "Apr 27 10:53" /var/log messages" ought to do the filtering...

[coolemail]

thank you Biggles.

There is nothing in var/log/messages to suggest that anything else happened to the rejected Spam email.

On that matter, can I add certain addresses to whitelist_senders which are then whitelisted on the checking? For exapmple, could I put in mailman-bounces@*.* in the whitelist_senders?

The other issue that led me to realise this was happening was because an email that should have been sent to a mailman mailing list owner for approval is not getting through. In /usr/local/psa/var/log/maillog, it shows the following:

Code: Select all

Apr 28 08:00:08 plesk2 spamdyke[3741]: ALLOWED from: coltsplayers-bounces@salisburyrfc.org to: coltsplayers-owner@salisburyrfc.org origin_ip: 127.0.0.1 origin_rdns: localhost auth: (unknown)
Apr 28 08:00:09 plesk2 qmail-queue[3753]: scan: the message(drweb.tmp.HqgthF) sent by coltsplayers-bounces@salisburyrfc.org to coltsplayers-owner@salisburyrfc.org should be passed without checks, because contains uncheckable addresses
Apr 28 08:00:09 plesk2 qmail-queue-handlers[3756]: from=coltsplayers-bounces@salisburyrfc.org
Apr 28 08:00:09 plesk2 qmail-queue-handlers[3756]: to=coltsplayers-owner@salisburyrfc.org
Apr 28 08:00:09 plesk2 qmail-queue-handlers[3756]: recipient[3] = 'coltsplayers-owner@salisburyrfc.org'
Apr 28 08:00:09 plesk2 qmail-queue-handlers[3756]: handlers dir = '/var/qmail//handlers/before-queue/recipient/coltsplayers-owner@salisburyrfc.org'
Apr 28 08:00:09 plesk2 qmail: 1240902009.031785 info msg 14814362: bytes 1203 from <coltsplayers-bounces@salisburyrfc.org> qp 3757 uid 2020
Apr 28 08:00:09 plesk2 qmail: 1240902009.045658 starting delivery 8328: msg 14814362 to local 65-coltsplayers-owner@salisburyrfc.org
Apr 28 08:00:09 plesk2 qmail-local-handlers[3758]: from=coltsplayers-bounces@salisburyrfc.org
Apr 28 08:00:09 plesk2 qmail-local-handlers[3758]: to=coltsplayers-owner@salisburyrfc.org
Apr 28 08:02:41 plesk2 spamdyke[4129]: DENIED_SENDER_NO_MX from: mailman-bounces@plesk2 to: coltsadmin@salisburyrfc.org origin_ip: 127.0.0.1 origin_rdns: localhost auth: (unknown)
Apr 28 08:02:51 plesk2 spamdyke[4136]: DENIED_SENDER_NO_MX from: mailman-bounces@plesk2 to: coltsadmin@salisburyrfc.org origin_ip: 127.0.0.1 origin_rdns: localhost auth: (unknown)
Apr 28 08:03:01 plesk2 spamdyke[4156]: DENIED_SENDER_NO_MX from: mailman-bounces@plesk2 to: coltsadmin@salisburyrfc.org origin_ip: 127.0.0.1 origin_rdns: localhost auth: (unknown)
Apr 28 08:03:11 plesk2 spamdyke[4170]: DENIED_SENDER_NO_MX from: mailman-bounces@plesk2 to: coltsadmin@salisburyrfc.org origin_ip: 127.0.0.1 origin_rdns: localhost auth: (unknown)
Apr 28 08:03:32 plesk2 spamdyke[4191]: DENIED_SENDER_NO_MX from: mailman-bounces@plesk2 to: coltsadmin@salisburyrfc.org origin_ip: 127.0.0.1 origin_rdns: localhost auth: (unknown)

and the email is being sent from mailman-bounces@plesk2 [mailto:mailman-bounces@plesk2] On Behalf Of coltsadmin-bounces@salisburyrfc.org, causing the DENIED_SENDER_NO_MX

I think it should be going from mailman-bounces@plesk2.expat-email.co.uk. Can you tell me what config file might hold this information so that I can get the moderator emails to be sent from the full address?

[faris]

Ah! Interesting!

Two things then:

1) Make sure the hostname for the server is set to plesk2.expat-email.co.uk

2) /var/qmail/control/me contains the domain name that qmail considers to be the server name. So again it should be plesk2.expat-email.co.uk

And also....
3) You may want to add 127.0.0.1 and also the public IP of your server to the ip whitelist file for spamdyke (then restart xinetd) . Doing so will allow all email from localhost/server's IP to bypass any spamdyke filtering. This is sometimes necessary.

Faris.

[coolemail]

1) Make sure the hostname for the server is set to plesk2.expat-email.co.uk

2) /var/qmail/control/me contains the domain name that qmail considers to be the server name. So again it should be plesk2.expat-email.co.uk

Both of the above were plesk2.expat-email.co.uk

3) You may want to add 127.0.0.1 and also the public IP of your server to the ip whitelist file for spamdyke (then restart xinetd) . Doing so will allow all email from localhost/server's IP to bypass any spamdyke filtering. This is sometimes necessary.

I have done that, and restarted xinetd. I have noticed 3 things:

1. Monitoring the maillog, I notice a known email address being rejected:

Code: Select all

Apr 28 12:23:13 plesk2 spamdyke[7079]: DENIED_SENDER_NO_MX from: someone@wiltscc.gov.uk to: email@hosted-domain.com origin_ip: 85.158.140.195 origin_rdns: mail193.messagelabs.com auth: (unknown)

. http://spamdyke.org/documentation/READM ... _ADDRESSES suggest that it is not good practice to put addresses in the sender file. And in any case, the filenames appear to be different to the ones that are in /var/qmail/spamdyke. What should I do for examples like this? And can I find out why they do not have an MX - I find it strange that a governemnt organisation does not have one?

2. More worrying, some outgoing mailman mailing list emails are not going with the problem in my last post:

Code: Select all

Connected to 194.154.164.152 but sender was rejected.
Remote host said: 550-Verification failed for <mailman-bounces@plesk2> 550-Unrouteable address 550 ATLAS(2503): Sender verification failed

.
and

Code: Select all

Connected to 193.252.22.141 but sender was rejected.
Remote host said: 504 5.5.2 <mailman-bounces@plesk2>: Sender address rejected: need fully-qualified address

This suggests that I MUST find the source of whatever it is which is sending mailing list emails from mailman-bounces@plesk2 and not mailman-bounces@plesk2.expat-email.co.uk.

3. In Plesk CP Server > Service Management, "SMTP Server (QMail)" is showing as running. But in a previous post, Faris suggested that with spamdyke running, qmail would show as not running when I reported that. Does that mean that it is not running properly? I don't think so because maillog is reporting spamdyke entries.

Really grateful if you could help me.

EDIT. Regarding point 2 above, the whole headers of the email are as follows, and it appears that it is putting the Return-Path as just plesk2

Hi. This is the qmail-send program at plesk2.expat-email.co.uk.
I tried to deliver a bounce message to this address, but the bounce bounced!

<mailman-bounces@plesk2>:
Sorry, I couldn't find any host named plesk2. (#5.1.2)

--- Below this line is the original bounce.

Return-Path: <>
Received: (qmail 8596 invoked for bounce); 28 Apr 2009 12:32:52 +0100
Date: 28 Apr 2009 12:32:52 +0100
From: MAILER-DAEMON@plesk2.expat-email.co.uk
To: mailman-bounces@plesk2
Subject: failure notice

Hi. This is the qmail-send program at plesk2.expat-email.co.uk.
I'm afraid I wasn't able to deliver your message to the following addresses.
This is a permanent error; I've given up. Sorry it didn't work out.

<email@external-domain.co.uk>: [this is the email forwarding address for coltsadmin@salisburyrfc.org - see below]
Connected to 194.154.164.158 but sender was rejected.
Remote host said: 550-Verification failed for <mailman-bounces@plesk2> 550-Unrouteable address 550 ATLAS(2503): Sender verification failed

--- Below this line is a copy of the message.

Return-Path: <mailman-bounces@plesk2>
Received: (qmail 8480 invoked by uid 110); 28 Apr 2009 12:32:42 +0100
Delivered-To: 65-coltsadmin@salisburyrfc.org
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
plesk2.expat-email.co.uk
X-Spam-Level:
X-Spam-Status: No, score=-1.4 required=4.0 tests=ALL_TRUSTED,AWL autolearn=ham
version=3.2.5
Received: (qmail 8348 invoked from network); 28 Apr 2009 12:32:38 +0100
Received: from localhost (HELO ?82.197.79.4?) (127.0.0.1)
by localhost with SMTP; 28 Apr 2009 12:32:38 +0100
Return-Path: <coltsplayers-bounces@salisburyrfc.org>
Delivered-To: 65-coltsplayers-owner@salisburyrfc.org
Received: (qmail 13481 invoked from network); 27 Apr 2009 08:00:09 +0100
Received: from localhost (HELO ?82.197.79.4?) (127.0.0.1)
by localhost with SMTP; 27 Apr 2009 08:00:09 +0100

[faris]

Number 2 sounds like a dkim issue, and if so will be unrelated to spamdyke/qmail but will be realted to the setup of domain keys in Plesk. Just a guess.

Talking of Plesk, is the hostname set correctly in the control panel itself (somewhere in the Server page, under options or edit or something like that?

Also try running the following on the command line.

hostname
hostname -f
hostname -d

The last one (hostname -d) should only return the last part of the hostname (i.e. excluding the plesk2 part).
The other two should give the whole thing.

You can check people's DNS and MX records using www.dnsstuff.com -- very handy. You should then compare the result with that you get when you use dig @localhost or nslookup (again using localhost) to make sure your machine is doing DNS lookups correctly.

There is a brilliant mailing list for spamdyke which you might want to join.
http://www.spamdyke.org/mailman/listinfo/spamdyke-users

[coolemail]

Thank you, as ever, faris,

1. The hostnames are all OK, and yes, the Plesk setup for the hostname is the same in CP >Server > Server Preferences

[root@plesk2 ~]# hostname
plesk2.expat-email.co.uk
[root@plesk2 ~]# hostname -f
plesk2.expat-email.co.uk
[root@plesk2 ~]# hostname -d
expat-email.co.uk
[root@plesk2 ~]#

I am registered with dnsstuff and use it loads. I agree with you that it is incredibly useful, and I often check the DNS records of people. It does not help me with why the mailman appears to be sending its messages with Return-Path: <mailman-bounces@plesk2> and not Return-Path: <mailman-bounces@plesk2.expat-email.co.uk> which I guess it should be instead?

I will join the spamdyke mailing list - I have been trawling the forums recently about this issue.

If there any way that I can find out what other mailman mailing list emails have been rejected by 3rd parties? The ones I have found are only coming up when a bounce bounced, or I happened to be monitoring the maillog. The grep is a useful tool if I suspect something specific, but that would not cover the historical mailman issues.

Many thanks for staying with me through this one, and I hope that someone will be able to help with how to finally resolve this one.

[faris]

Interesting! I don't get it. Most mysterious.

Isn't there some sort of configuration option for Mailman in Plesk? Could some kind of hostname or email address be set in that? I seem to remember that it gets grayed out after you configure it for the first time (really useful!). You can re-activate it by....errr...deleting a file. I don't remember which file. Is there an /etc/mailman or something?

I notice that the Parallels Plesk forums are now a little more active after they make it possible to find new posts that didn't involve a million Parallels Desktop home users bleating about their usb devices not working or complaining that they had not realised they had to actually BUY a copy of Windows XP in order to run that OS on their Mac and the like....

So maybe it would be worth posting there in order to reach a wider audience.

Faris.

[coolemail]

Thank you faris,

I have been so frustrated with the Plesk forum. Everything has to be moderated first before posting, and so much goes unanswered. However, I will trawl that again, and go to the mailman forum as well to see if they can help. If I get an answer, I will certainly post it here!

Incidentally, i would like to be able to give some feedback to people on some of the rejections that are happening. I noticed the following one:

Apr 29 17:29:56 plesk2 spamdyke[3943]: DENIED_RDNS_RESOLVE from: email1@strathallan.co.uk to: knownaddress@hosteddomain.com origin_ip: 195.173.86.10 origin_rdns: no-dns-yet.demon.co.uk auth: (unknown)

So i tried to get some more information on it. DNS Tools does not come up with anything of any concern
http://private.dnsstuff.com/tools/dnsre ... 963954f019
with the following on MX:

Your 2 MX records are:

10 mxhost-2.vi.net. [TTL=86400] IP=212.78.66.155 [TTL=600] [GB]
IP=212.78.66.158 [TTL=600] [GB]
IP=212.78.66.187 [TTL=600] [GB]
IP=212.78.66.188 [TTL=600] [GB]
IP=212.78.66.189 [TTL=600] [GB]
6 webmail.strathallan.co.uk. [TTL=86400] IP=195.173.86.12 [TTL=86400] [GB]

So then i did the following.

[root@plesk2 ~]# dig mx strathallan.co.uk

; <<>> DiG 9.3.4-P1 <<>> mx strathallan.co.uk
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17403
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 2, ADDITIONAL: 3

;; QUESTION SECTION:
;strathallan.co.uk. IN MX

;; ANSWER SECTION:
strathallan.co.uk. 21745 IN MX 6 webmail.strathallan.co.uk.
strathallan.co.uk. 21745 IN MX 10 mxhost-2.vi.net.

;; AUTHORITY SECTION:
strathallan.co.uk. 21745 IN NS ns5.vi.net.
strathallan.co.uk. 21745 IN NS ns4.vi.net.

;; ADDITIONAL SECTION:
webmail.strathallan.co.uk. 21745 IN A 195.173.86.12
ns4.vi.net. 108145 IN A 212.78.66.36
ns5.vi.net. 108145 IN A 212.78.64.69

;; Query time: 7 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Apr 30 12:00:13 2009
;; MSG SIZE rcvd: 174

[root@plesk2 ~]# host -r 195.173.86.10
10.86.173.195.in-addr.arpa has no PTR record
[root@plesk2 ~]# host strathallan.co.uk
strathallan.co.uk mail is handled by 10 mxhost-2.vi.net.
strathallan.co.uk mail is handled by 6 webmail.strathallan.co.uk.
[root@plesk2 ~]#

Is it the lack of a PTR record which caused it to fail with the DENIED_RDNS_RESOLVE? And what advice would you send back to the email address on my server that they were sending to, if not to tell them to go back to the sender and tell them that they should sort out the PTR - and switch hosting to me!!

[faris]

Its a lack of A record on no-dns-yet.demon.co.uk that's the problem here. Not a lack of PTR.

RDNS_RESOLVE means the rdns for the IP does not resolve to an IP (any IP - does not have to be the same IP).

So Spamdyke is saying that 195.173.86.10 resolves to no-dns-yet.demon.co.uk (so there is a PTR for that IP -- I checked and it definitely has one) but no-dns-yet.demon.co.uk does not resolve to anything (so no A record - again I checked and it is true).

This will result in a rejection by spamdyke if you have the appropriate setting enabled (which you do, and most people do).

The only false positives (if you can even call them that) we've ever seen are from Demon. So you could whitelist that domain if you wanted to. We have not done so, however, and don't intend to.

Faris.