SpamAssassin USER_IN_WHITELIST

dietcheese · Unread post by **dietcheese** » Thu Feb 21, 2008 2:06 pm

Hi all,

I'm daily getting 4 or 5 of these spam e-mails that are making it past spamassassin/qmail-scanner with a SA:0(-75.3/8.0) rating.

The headers are:

Code: Select all

(spamassassin: 3.2.3. perlscan: 2.01st.
Clear:RC:1(127.0.0.1):.
Processed in 0.039784 secs); 21 Feb 2008 14:31:02 -0000
Delivered-To: 125-me@myserver.com
Received: (qmail 13220 invoked by uid 10064); 21 Feb 2008 09:31:01 -0500
Received: from 92.112.126.145 by 109839-app1.myserver.com (envelope-from <chad@lifesucks.co.uk>, uid 2020) with qmail-scanner-2.01st
(spamassassin: 3.2.3. perlscan: 2.01st.
Clear:RC:0(92.112.126.145):SA:0(-75.3/8.0):.
Processed in 1.088771 secs); 21 Feb 2008 14:31:01 -0000
X-Spam-Status: No, hits=-75.3 required=8.0
X-Qmail-Scanner-Mail-From: chad@lifesucks.co.uk via 109839-app1.myserver.com
X-Qmail-Scanner: 2.01st (Clear:RC:0(92.112.126.145):SA:0(-75.3/8.0):. Processed in 1.088771 secs Process 13206)
Received: from 145-126-112-92.pool.ukrtel.net (HELO pari-to6nkbm3hg) (92.112.126.145)
by 109839-app1.myserver.com with SMTP; 21 Feb 2008 09:31:00 -0500
Received-SPF: softfail (109839-app1.myserver.com: transitioning SPF record at mailanyone.net does not designate 92.112.126.145 as permitted sender)
Content-Return: allowed
X-Mailer: CME-V6.5.4.3; MSN
Return-Path: communications_msn_cs_enus@cimail15.msn.com
Received: (qmail 2224 by uid 682); Thu, 21 Feb 2008 04:31:05 +0200
Message-Id: <20080221063105.2226.qmail@pari-to6nkbm3hg>
To: <me@myserver.com>
Subject: January 72% OFF
From: <me@myserver.com>
MIME-Version: 1.0
Content-Type: text/html; charset="ISO-8859-1"
Content-Transfer-Encoding: 7bit

145-126-112-92.pool.ukrtel.net is a dynamic IP in the Ukraine...so I'm not sure why SpamAssassin seems to be whitelisting their address:

Code: Select all

Feb 21 09:31:01 109839-app1 spamd[580]: spamd: clean message (-75.3/8.0) for qscand:10064 in 0.9 seconds, 3571 bytes.
Feb 21 09:31:01 109839-app1 spamd[580]: spamd: result: . -75 - AWL,BAYES_99,DCC_CHECK,DYN_RDNS_SHORT_HELO_HTML,HTML_IMAGE_ONLY_32,HTML_MESSAGE,MIME_HTML_ONLY,MISSING_DATE,RCVD
_IN_PBL,RDNS_DYNAMIC,TVD_RCVD_IP,URIBL_AB_SURBL,URIBL_BLACK,URIBL_JP_SURBL,URIBL_OB_SURBL,URIBL_SBL,URIBL_SC_SURBL,URIBL_WS_SURBL,USER_IN_WHITELIST scantime=0.9,size=3571,user
=qscand,uid=10064,required_score=8.0,rhost=localhost,raddr=127.0.0.1,rport=47095,mid=<20080221063105.2226.qmail@pari-to6nkbm3hg>,bayes=0.994933,autolearn=no

I even tried su-ing to qscand and running these headers through spamassassin which did not result in the message being whitelisted.

Any ideas?

Maybe qmail-scanner is mangling the headers before passing it to SpamAssassin or something? How can I test and/or fix this?

Thanks,
DC

Unread post by **scott** » Thu Feb 21, 2008 2:58 pm

That means you've got that email address, or some wildcard of it in a spamassassin whitelist.

dietcheese · Unread post by **dietcheese** » Thu Feb 21, 2008 3:20 pm

Hmm...I didn't put it there and nobody else has access.

Specific files I should look at?

I appreciate your help.

DC

Unread post by **scott** » Thu Feb 21, 2008 11:54 pm

look at the spamassassin .cf files or any user prefs files out there on the box

dietcheese · Unread post by **dietcheese** » Fri Feb 22, 2008 12:54 am

I did a whole slew of recursive greps on my .cf files (don't have individual user_pref's set up). I searched on the entire IP, the class A and B numbers, and the domain 'ukrtel'. I came up with nothing.

I just received another one of these e-mails that made it past spamassassin. The body is the same as the previous one but the headers are somewhat different:

Code: Select all

Return-Path: <david.toralez@liebert.com>
Delivered-To: 125-me@myserver.com
Received: (qmail 7581 invoked by uid 10064); 21 Feb 2008 23:18:06 -0500
Received: from 127.0.0.1 by 109839-app1.myserver.com (envelope-from <david.toralez@liebert.com>, uid 110) with qmail-scanner-2.01st 
 (spamassassin: 3.2.3. perlscan: 2.01st.  
 Clear:RC:1(127.0.0.1):. 
 Processed in 0.027778 secs); 22 Feb 2008 04:18:06 -0000
Delivered-To: 125-me@myserver.com
Received: (qmail 7566 invoked by uid 10064); 21 Feb 2008 23:18:06 -0500
Received: from 221.219.33.67 by 109839-app1.myserver.com (envelope-from <david.toralez@liebert.com>, uid 2020) with qmail-scanner-2.01st 
 (spamassassin: 3.2.3. perlscan: 2.01st.  
 Clear:RC:0(221.219.33.67):SA:0(-72.8/8.0):. 
 Processed in 0.931805 secs); 22 Feb 2008 04:18:05 -0000
X-Spam-Status: No, hits=-72.8 required=8.0
X-Qmail-Scanner-Mail-From: david.toralez@liebert.com via 109839-app1.myserver.com
X-Qmail-Scanner: 2.01st (Clear:RC:0(221.219.33.67):SA:0(-72.8/8.0):. Processed in 0.931805 secs Process 7550)
Received: from unknown (HELO 64F42E4D9A1D414) (221.219.33.67)
  by 109839-app1.myserver.com with SMTP; 21 Feb 2008 23:18:04 -0500
Received-SPF: softfail (109839-app1.myserver.com: transitioning SPF record at liebert.com does not designate 221.219.33.67 as permitted sender)
X-Mailer: CME-V6.5.4.3; MSN 
Return-Path: communications_msn_cs_enus@cimail15.msn.com
Received: (qmail 3805 by uid 409); Fri, 22 Feb 2008 12:18:07 +0800
Message-Id: <20080222201807.3807.qmail@64F42E4D9A1D414>
To: <me@myserver.com>
Subject: February 77% OFF
From: <me@myserver.com>
MIME-Version: 1.0
Content-Type: text/html; charset="ISO-8859-1"
Content-Transfer-Encoding: 7bit

I'm really at a loss here. Anyone have any ideas?

Maybe I'm grepping for the wrong thing?

I appreciate it,
DC

dietcheese · Unread post by **dietcheese** » Sat Feb 23, 2008 11:19 pm

Still no solution to that last problem...I have spent days and cannot seem to fix it.

Now I'm getting a second problem trying to receive e-mails from Freshbooks. Every time an invoice comes to me I get:

Code: Select all

Feb 23 22:11:53 109839-app1 qmail-scanner[31079]: Policy:Bad_MIME_Type:RC:0(72.32.48.26):SA:0(-106.6/8.0): 1.904444 984 admin@blahblah.com me@mydomain.com
New_invoice_from_BlahBlah<20080224031151.AB87F1E0F95@server1.freshbooks.com> 1203822711.31081-0.109839-app1.myserver.com:334

-106 !?

I am beginning to think my qmail-scanner is hosed.

Any idea what this Bad_MIME_Type refers to? Not much on google. I've tried reconfiguring qmail-scanner, playing with the FIX_MIME setting - to no avail.

It's strange - SA tags it as -106 but I never get the e-mail, nor is it quarantined. I figure qmail-scanner is eating it but I'm not sure where to look.

Any help would be appreciated.

Thanks!
DC

dietcheese · Unread post by **dietcheese** » Sat Feb 23, 2008 11:43 pm

Just for the record I changed BAD_MIME_MIME checks to 2 in qmail-scanner-queue.

my $BAD_MIME_CHECKS='2';

Which let the Freshbooks e-mails through.

Is something noticeably wrong with their e-mail headers?

Code: Select all

Return-Path: <admin@myserver.com>
Delivered-To: 124-me@foo.tv
Received: (qmail 5268 invoked by uid 10064); 23 Feb 2008 22:34:39 -0500
Received: from 72.32.48.26 by 109839-app1.myserver.com (envelope-from <admin@myserver.com>, uid 2020) with qmail-scanner-2.01st 
 (spamassassin: 3.2.4. perlscan: 2.01st.  
 Clear:RC:0(72.32.48.26):SA:0(-6.6/8.0):. 
 Processed in 1.189272 secs); 24 Feb 2008 03:34:39 -0000
X-Spam-Status: No, hits=-6.6 required=8.0
X-Qmail-Scanner-Mail-From: admin@myserver.com via 109839-app1.myserver.com
X-Qmail-Scanner: 2.01st (Clear:RC:0(72.32.48.26):SA:0(-6.6/8.0):. Processed in 1.189272 secs Process 5256)
Received: from server1.freshbooks.com (72.32.48.26)
  by 109839-app1.myserver.com with SMTP; 23 Feb 2008 22:34:38 -0500
Received-SPF: unknown (109839-app1.myserver.com: Multiple SPF records returned)
Received: by server1.freshbooks.com (Postfix, from userid 48)
	id 2200E1E10E1; Sat, 23 Feb 2008 22:34:38 -0500 (EST)
To: me@foo.tv
Subject: New invoice from My Company
MIME-Version: 1.0
From: admin@myserver.com
Reply-to: admin@myserver.com
X-Priority: 3
X-Mailer: PHP mailer
Message-Id: <20080224033438.2200E1E10E1@server1.freshbooks.com>
Date: Sat, 23 Feb 2008 22:34:38 -0500 (EST)
X-Qmail-Scanner-2.01st: added fake Content-Type header
Content-Type: text/plain

Anyone know what this:

X-Qmail-Scanner-2.01st: added fake Content-Type header

Is all about?

Unread post by **scott** » Sun Feb 24, 2008 10:15 am

That means that Freshbooks fails at coding.

They arent generating RFC correct MIME headers.

dietcheese · Unread post by **dietcheese** » Sun Feb 24, 2008 12:49 pm

Does it mean they didn't include a content-type header and so qmail-scanner appended a text/plain to their headers or that they did something else incorrectly?

Unread post by **scott** » Mon Feb 25, 2008 8:07 am

Right, it will try to correct the problem if it can. There are a lot of different cases where it won't be able to, which is probably whats happening in your example.

dietcheese · Unread post by **dietcheese** » Mon Feb 25, 2008 11:36 am

Thanks Scott -

Quick question. I have been reading some of the RFC's but I'm a bit confused. Is the content-type header actually required or just suggested?

It seems to me, but I'm not sure, that it is only required if the MIME-Version header is defined. If MIME-Version is not defined then content-type is not required and is assumed to be text/plain.

My brain can't parse the RFCs right now.

Thanks for all your help and patience...

DC

dietcheese · Unread post by **dietcheese** » Mon Feb 25, 2008 6:08 pm

I dunno. Here's what the RFC says

5.2. Content-Type Defaults

Default RFC 822 messages without a MIME Content-Type header are taken
by this protocol to be plain text in the US-ASCII character set,
which can be explicitly specified as:

Content-type: text/plain; charset=us-ascii

This default is assumed if no Content-Type header field is specified.
It is also recommend that this default be assumed when a
syntactically invalid Content-Type header field is encountered. In
the presence of a MIME-Version header field and the absence of any
Content-Type header field, a receiving User Agent can also assume
that plain US-ASCII text was the sender's intent. Plain US-ASCII
text may still be assumed in the absence of a MIME-Version or the
presence of an syntactically invalid Content-Type header field, but
the sender's intent might have been otherwise.

Freshbooks should be OK not having a content-type as far as I can tell.

Maybe qmail-scanner is wrong. Or is it choking on something else besides a missing content type? If a content-type is missing, and it is RFC compliant in that sense, then shouldn't qmail just assume text/plain and then send it thru.

I'm really confused.

DC

dietcheese · Unread post by **dietcheese** » Mon Feb 25, 2008 9:56 pm

Anyone have any ideas on this?

Here's the quarantined e-mail:

Received: from server1.freshbooks.com (72.32.48.26)
by 109839-app1.myserver.com with SMTP; 25 Feb 2008 20:37:28 -0500Received: by server1.freshbooks.com (Postfix, from userid 48)
id BAE111E0BA0; Mon, 25 Feb 2008 20:37:27 -0500 (EST)To: me@foo.tv
Subject: New invoice from Warx, Inc.
MIME-Version: 1.0
From: admin@myserver.com
Reply-to: admin@myserver.com
X-Priority: 3
X-Mailer: PHP mailer
Message-Id: <20080226013727.BAE111E0BA0@server1.freshbooks.com>
Date: Mon, 25 Feb 2008 20:37:27 -0500 (EST)

John Smith

To access your invoice, please go to:

https://domain.freshbooks.com/inv/54612-3

Thank you,

Warx, Inc
admin@domain.com

*** Qmail-Scanner Quarantine Envelope Details Begin ***
X-Qmail-Scanner-Mail-From: "admin@myserver.com" via 109839-app1.myserver.com
X-Qmail-Scanner-Rcpt-To: "me@foo.tv"
X-Qmail-Scanner: 2.01st (spamassassin: 3.2.4. perlscan: 2.01st. policy-violation Found. Processed in 1.139877 secs) process 17798
Quarantine-Description: Disallowed MIME Content-Type found - not valid email
*** Qmail-Scanner Envelope Details End ***

Qmail is saying:

Quarantine-Description: Disallowed MIME Content-Type found - not valid email

Presumably because there is no content-type in the e-mail.

However, RFC 2045 says:

5.2. Content-Type Defaults

Default RFC 822 messages without a MIME Content-Type header are taken
by this protocol to be plain text in the US-ASCII character set,
which can be explicitly specified as:

Content-type: text/plain; charset=us-ascii

This default is assumed if no Content-Type header field is specified.
It is also recommend that this default be assumed when a
syntactically invalid Content-Type header field is encountered. In
the presence of a MIME-Version header field and the absence of any
Content-Type header field, a receiving User Agent can also assume
that plain US-ASCII text was the sender's intent. Plain US-ASCII
text may still be assumed in the absence of a MIME-Version or the
presence of an syntactically invalid Content-Type header field, but
the sender's intent might have been otherwise.

Is qmail-scanner wrong in calling these e-mail headers invalid because they lack a content-type?

Thanks,
DC

dietcheese · Unread post by **dietcheese** » Mon Feb 25, 2008 10:09 pm

I wonder if this qmail-scanner-queue.pl code is wrong:

Code: Select all

  if ( ($BAD_MIME_CHECKS > 1 && $headers{'mime-version'} eq "") || ($headers{'mime-version'} ne "" && $headers{'content-type'} =~ /^text\/plain/i)) {
    #Hmm, doesn't look nice, but it feels better to make this a separate check for some reason
    if ($skip_text_msgs && ($indicates_attachments < 2) && !@uufile_list && !@attachment_list) {
      &debug("This is a PLAIN text message (because it's either not mime, or is text/plain), skip virus scanners - but not antispam scanners");
      &minidebug("This is a PLAIN text message, skip virus scanners - but not SA");
      $plain_text_msg=1;
    }
  }

It seems to try and make an e-mail "plain text" if:

1) there is no MIME-Version listed or
2) there IS a MIME-Version listed AND content-type='text/plain'

But doesn't account for

3) there IS a MIME-Version listed AND no content-type

Which is perfectly valid.

My perl is pretty rusty and i haven't gone through the rest of the code though...

DC

dietcheese · Unread post by **dietcheese** » Wed Feb 27, 2008 2:07 pm

anybody?

Atomicorp

SpamAssassin USER_IN_WHITELIST

SpamAssassin USER_IN_WHITELIST

no luck

Freshbooks