Why are "allowed" packets being logged?

User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: Why are "allowed" packets being logged?

Unread post by mikeshinn »

Rules look fine. So this:
Apr 16 12:07:48 server7 kernel: DROP_ASL_INPUT IN=eth0 OUT= MAC=(redacted) SRC=MyIP DST=serverIP LEN=40 TOS=0x00 PREC=0x00 TTL=58 ID=37088 DF PROTO=TCP SPT=62996 DPT=30000 SEQ=4005276762 ACK=0 WINDOW=0 RES=0x00 RST URGP=0
Means your client is aborting, not closing the connection, and the connection is already closed, which is why the packet is dropped.
biggles
Forum Regular
Forum Regular
Posts: 806
Joined: Tue Jul 15, 2008 2:38 pm
Location: Sweden
Contact:

Re: Why are "allowed" packets being logged?

Unread post by biggles »

OK. Might I suggest there might be something erroneous with the web gui? I use a standard Chrome client on OS X most of the time. Or maybe the detection needs to be adjusted, because just using the web gui isn't suppose to trigger an block, is it?
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: Why are "allowed" packets being logged?

Unread post by mikeshinn »

OK. Might I suggest there might be something erroneous with the web gui? I use a standard Chrome client on OS X most of the time. Or maybe the detection needs to be adjusted, because just using the web gui isn't suppose to trigger an block, is it?
ASL wont block on a single firewall event, if you are getting shunned that means one of three things is happening:

1) invalid packet dropping is not configured (https://www.atomicorp.com/wiki/index.ph ... OP_INVALID)
2) somehow the firewall rules are out of order
3) the client is sending a LOT of RST packets, way way too many - which can happen is the client has a broken firewall (on the client) or a broken stack/client
4) the system isnt using the ASL kernel so it doesnt support INVALID tracking and is treating invalid packets as valid packets

The client, not the server, is sending all these packets, and ASL wont block on a single firewall event. So if you are getting a firewall shun, and if FW_DROP_INVALID is enabled then the client is doing something really strange and its sending blind RSTs to the server. And to be honest I've never seen a client do this unless either its stack is broken, the client has a broken firewall/accelerator or the client itself is broken.

One last thing that could be happening is if you setup a custom rule to allow connections to port 30000, or your rules are somehow out of order you could end up with a case where INVALID checks dont occur on port 30000. The order of your rules should look like this:

iptables -L -n

Chain INPUT (policy ACCEPT)

First block are the active response rules:

ASL-ACTIVE-RESPONSE all -- 1.2.3.4 0.0.0.0/0

Then geoblocking:

ASL-GEO-BLACKLIST all -- 0.0.0.0/0 0.0.0.0/0

Then the blacklists:

ASL-BLACKLIST all -- 0.0.0.0/0 0.0.0.0/0

If you have it enabled, small packet blocking:

ASL-SMALLPACKETS ah -- 0.0.0.0/0 0.0.0.0/0 length 0:35
ASL-SMALLPACKETS esp -- 0.0.0.0/0 0.0.0.0/0 length 0:49
ASL-SMALLPACKETS 47 -- 0.0.0.0/0 0.0.0.0/0 length 0:39
ASL-SMALLPACKETS 30 -- 0.0.0.0/0 0.0.0.0/0 length 0:31
ASL-SMALLPACKETS icmp -- 0.0.0.0/0 0.0.0.0/0 length 0:27
ASL-SMALLPACKETS tcp -- 0.0.0.0/0 0.0.0.0/0 length 0:39
ASL-SMALLPACKETS udp -- 0.0.0.0/0 0.0.0.0/0 length 0:27
ASL-BADPACKETS tcp -- 0.0.0.0/0 0.0.0.0/0 tcp option=128
ASL-BADPACKETS tcp -- 0.0.0.0/0 0.0.0.0/0 tcp option=64

Then portscans:

ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x37
ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x2B
ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x29
ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x1A
ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x0A
ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x0D
ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x1C
ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x03
ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x00
ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x3F
ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x3F
ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x00
ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x01
ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x29
ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x29/0x29
ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x22/0x22
ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x11/0x01
ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x18/0x08
ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x06/0x06
ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x05/0x05
ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x03/0x03

Then, if you have it enabled, fragment dropping:

ASL-FRAGMENTS all -f 0.0.0.0/0 0.0.0.0/0

Then invalid packet dropping:

DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID

And now all the rules that allow traffic:

This is a special loopback rule for the ASL web console:

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:30000 state NEW

This is the tortixd ACL (if you use this, this allows you to limit the IPs that connect to the ASL web console):

ASL-TORTIXD-ACL tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:30000 state NEW

This is where you would see any of the experimental ACL rules (this is an alpha feature, dont worry if you dont see this, it means you arent experimenting)

LOCAL-0-port-ACL tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:port state NEW

And finally the main unless allowed/deny ruleset that generates the DROP_ASL_INPUT messages.

ASL-Firewall-INPUT all -- 0.0.0.0/0 0.0.0.0/0

The point being that INVALID needs to come ahead of any allowed rules, so if thats not the case on the system and an allowed rule comes before INVALID, then INVALID packets wont get dropped and will get treated as potentially malicious (if they arent valid, then they will be treated an attempt to probe the system).
prupert
Forum Regular
Forum Regular
Posts: 573
Joined: Tue Aug 01, 2006 2:45 pm
Location: Netherlands

Re: Why are "allowed" packets being logged?

Unread post by prupert »

That is a terrific post Mike. I think many ASL users - including me - find your posts invaluable.

One thing though, you note that if you are not running the ASL kernel there is no support for INVALID tracking. I always thought that most common recent kernels do support this properly, for example the stock CentOS 5/6 kernels. Care to confirm? :-)
Lemonbit Internet Dedicated Server Management
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: Why are "allowed" packets being logged?

Unread post by mikeshinn »

One thing though, you note that if you are not running the ASL kernel there is no support for INVALID tracking. I always thought that most common recent kernels do support this properly, for example the stock CentOS 5/6 kernels. Care to confirm? :-)
Thanks for the question.

INVALID may or may not exist in other kernels. If its in the default kernel, YMMV, it should work. If it doesnt, contact those vendors (we cant fix their kernels).

With that said, we know that its not always included. For example, we know that some versions of virtuzzo do not include it. Some hosting companies build their own kernels and invalid is a module thats not enabled by default, so we have seen custom built kernels that missed including it.

Now if you use our kernel, we know its there and we know it works.
biggles
Forum Regular
Forum Regular
Posts: 806
Joined: Tue Jul 15, 2008 2:38 pm
Location: Sweden
Contact:

Re: Why are "allowed" packets being logged?

Unread post by biggles »

Thanks a lot for your detailed answer Mike!

1. INVALID is set to "yes".

Code: Select all

 ASL firewall
FW_INBOUND_TCP_SERVICES="20,21,22,25,53,80,443,465,110,143,993,995,587,8443,10050,161,30000"
FW_INBOUND_UDP_SERVICES="53,123"
FW_OUTPUT_MTA="no"
FW_OUTPUT_TCP_SERVICES="22,25,80,443,465,123,53,10051"
FW_OUTPUT_UDP_SERVICES="53,123"
FW_LASSO="no"
FW_LASSO_LOG="yes"
FW_DSHIELD="no"
FW_DSHIELD_LOG="yes"
FW_TOR="no"
FW_TOR_LOG="yes"
FW_PORTSCAN="yes"
FW_BAD_PACKETS="yes"
FW_SMALL_PACKETS="no"
FW_FRAGMENTS="no"
FW_DROP_INVALID="yes"
FW_DROP_INVALID_LOG="no"
FW_LOG_BLACKLIST_DROP="yes"
FW_LOG_GEOBLOCK_DROP="yes"
FW_IGNORE_BROADCASTS="no"
FW_ACCEPT_REDIRECTS="no"
FW_ACCEPT_SOURCE_ROUTE="no"
FW_ICMP_IGNORE_ALL="no"
FW_ICMP_IGNORE_BROADCASTS="no"
FW_IGNORE_ICMP_BOGUS="no"
FW_IPV4_FORWARD="no"
FW_IPV6_FORWARD="no"
FW_PROXY_ARP="no"
FW_RP_FILTER="no"
FW_SYN_COOKIES="yes"
FW_TCP_ECN="no"
FW_TCP_TIMESTAMPS="yes"
FW_TCP_WINDOW_SCALING="yes"
FW_PLESK_UPDATES="yes"
FW_SPAMASSASSIN_UPDATES="yes"
2. Firewall rules look ok, according to your setup. You can also view them in an earlier post from me.
3. It cannot be a local router or firewall. It happens from mobile connections, from my home connection and from my office connection. Can a standard MacBook have this error in it's network stack?
4. I am using the ASL kernel

Might I add a 5? Could the extreme performance problems I'm seeing have anything to do with it? Could the client try ti reconnect several times due to the extrem long response time from the server?

As you can see I am using the simple firewall setup. As you also can see I have inserted port 30000 in the setup. This is necessary for me to be able to connect to the ASL GUI. I read in another port that it would be included by default, but it isn't for my servers at least.
prupert
Forum Regular
Forum Regular
Posts: 573
Joined: Tue Aug 01, 2006 2:45 pm
Location: Netherlands

Re: Why are "allowed" packets being logged?

Unread post by prupert »

Of course we would like to have the ASL kernel on all systems, but unfortunately have encountered several machines that did not want to boot with the ASL kernel. In that case we have no choice, but to use another kernel. For optimal support we pick the stock EL kernel.
mikeshinn wrote:INVALID may or may not exist in other kernels. If its in the default kernel, YMMV, it should work. If it doesnt, contact those vendors (we cant fix their kernels).
Default CentOS 5 kernel on a dedicated system, I presume this has proper support for a stateful firewall and detecting invalid packets? Because we see other modules on the ASL kernel, and because I regard you as the firewall guru on this forum, I'd like to ask once more to confirm that. (I do understand that you can't give support on non-ASL kernels, I am only asking to check...)

Code: Select all

# uname -svr
Linux 2.6.18-348.4.1.el5 #1 SMP Tue Apr 16 15:40:06 EDT 2013
# lsmod | grep conntrack
xt_conntrack           35649  0
ip_conntrack           92005  6 xt_connmark,xt_connlimit,xt_state,xt_conntrack,iptable_nat,ip_nat
nfnetlink              40457  2 ip_nat,ip_conntrack
x_tables               50505  30 ip6t_REJECT,ip6_tables,ipt_LOG,ipt_ecn,ipt_ECN,xt_string,xt_connmark,
    xt_connbytes,xt_connlimit,xt_tcpmss,xt_dscp,xt_DSCP,xt_mark,xt_MARK,xt_multiport,xt_pkttype,
    xt_quota,xt_physdev,xt_mac,xt_limit,xt_length,xt_state,xt_conntrack,xt_tcpudp,ipt_REDIRECT,
    ipt_recent,ipt_REJECT,ipt_owner,iptable_nat,ip_tables
(line-breaks added for readability)

ASL kernel has the following:

Code: Select all

# uname -srv
Linux 2.6.32.60-39.art.x86_64 #1 SMP Sat Mar 16 20:21:07 EDT 2013
# lsmod | grep conntrack
nf_conntrack_sane       6318  0
nf_conntrack_ftp       12591  1 nf_nat_ftp
nf_conntrack_netbios_ns     2350  0
Lemonbit Internet Dedicated Server Management
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: Why are "allowed" packets being logged?

Unread post by mikeshinn »

Might I add a 5? Could the extreme performance problems I'm seeing have anything to do with it? Could the client try ti reconnect several times due to the extrem long response time from the server?
Potentially. If the client just gives up and sends a RST, which is the client saying to abort the connection I suppose that could happen. Keep in mind its the client sending the RST packets, to a socket thats already closed. You could try disabling blackholing if your client really needs to do this:

https://www.atomicorp.com/wiki/index.ph ... _BLACKHOLE

Keep in mind that if the kernel is locked, you'll need to reboot for this to take effect.
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: Why are "allowed" packets being logged?

Unread post by mikeshinn »

Default CentOS 5 kernel on a dedicated system, I presume this has proper support for a stateful firewall and detecting invalid packets? Because we see other modules on the ASL kernel, and because I regard you as the firewall guru on this forum, I'd like to ask once more to confirm that. (I do understand that you can't give support on non-ASL kernels, I am only asking to check...)
I can't answer that. Thats not our kernel, you'll want to ask the Centos team that question. I imagine it should, but its not ours so I can't tell you for sure, we do not support that kernel nor do we use it.
Post Reply