Page 2 of 5

Re: nf_conntrack: table full, dropping packet

Posted: Thu Nov 25, 2010 10:16 am
by BruceLee
sqlite was discussed here somewhere.

Re: nf_conntrack: table full, dropping packet

Posted: Thu Nov 25, 2010 10:18 am
by chrismcb
Yep - got it.

To do with Sitebuilder (which i don't have).

Once that was removed, everything updated fine.


Thanks!

Re: nf_conntrack: table full, dropping packet

Posted: Thu Nov 25, 2010 10:48 am
by chrismcb
OK, last hurdle - i think (hope!).

When i first installed ASL, I stupidly gave it an admin username which I hadn't yet created.

As a result, it said it wasnt going to configure SSH - which i thought was fine.


I then created the user and reran asl -s -f, but this error is still appearing.

Code: Select all

 Checking Admin users
    Checking [user] directory /home/[user]: found      [OK]
    Checking [user] authorized_keys: not found           [FAILED]
    Valid Admin users detected: no                         [HIGH]
    WARNING: SSH will not be reconfigured at this time.
Can you advise how to "reconfigure" SSH?
I've tried everything I can think of - bar re-installing ASL!

Re: nf_conntrack: table full, dropping packet

Posted: Thu Nov 25, 2010 11:02 am
by BruceLee

Re: nf_conntrack: table full, dropping packet

Posted: Thu Nov 25, 2010 11:10 am
by chrismcb
Thanks, I've already done that and I am looging in with a key instead of password.

It just seems as if ASL doesnt recognise that because the user was created after the install.


Is there any harm in trying a re-install of ASL through the sh script?

Re: nf_conntrack: table full, dropping packet

Posted: Thu Nov 25, 2010 11:27 am
by scott
You just need to define the user in /etc/asl/config under "ADMIN_USERS". Thats a safety check to keep you from locking yourself out if you dont have valid keys.

Re: nf_conntrack: table full, dropping packet

Posted: Thu Nov 25, 2010 11:31 am
by chrismcb
Thanks, but the username is already in there!

Code: Select all

NOTIFY="yes"
EMAIL="xxxx"
HOSTNAME="xxxx"
ADMIN_USERS="xxxx"
IP_WHITELIST="/etc/asl/whitelist"
SYSTEM_TYPE="webserver"
AUTOMATIC_UPDATES="daily"
UPDATE_TYPE="all"
RESTART_APACHE="yes"
APACHE_RESTART_COMMAND="/etc/init.d/httpd restart"
ASL_USER="tortix"
I have removed it from above, but I can confirm it is spelled correctly etc and the user is valid - it's the one i log in with before SU-ing to root!

Re: nf_conntrack: table full, dropping packet

Posted: Thu Nov 25, 2010 11:34 am
by scott
It couldnt find your key then

Re: nf_conntrack: table full, dropping packet

Posted: Thu Nov 25, 2010 11:43 am
by chrismcb
The key is located in:

Code: Select all

/home/xxx/.ssh/
It has the filename:

Code: Select all

authorized_keys2
And is owned by that user with CHMOD set to 600.


I believe that to be correct?

Re: nf_conntrack: table full, dropping packet

Posted: Thu Nov 25, 2010 11:53 am
by scott
No its not, that is deprecated. The file is :

~/.ssh/authorized_keys

Re: nf_conntrack: table full, dropping packet

Posted: Fri Nov 26, 2010 8:47 am
by chrismcb
This seemed to have cleared itself up after a couple of reboots!

Back up and running now on the new server (a late night/early morning for me last night) and just going through some final tweaking now.



Thanks everyone

Re: nf_conntrack: table full, dropping packet

Posted: Tue Nov 30, 2010 7:04 am
by chrismcb
OK, so it looks like this original problem was not related to the previous server attack - it's happening again.
Nov 30 10:50:09 xxxxx kernel: nf_conntrack: table full, dropping packet.
Nov 30 10:50:13 xxxxx kernel: nf_conntrack: table full, dropping packet.
Nov 30 10:50:52 xxxxx last message repeated 3 times
Nov 30 10:51:04 xxxxx last message repeated 3 times
Nov 30 10:51:16 xxxxx kernel: nf_conntrack: table full, dropping packet.
Nov 30 10:51:19 xxxxx last message repeated 3 times
Nov 30 10:51:25 xxxxx clamd[19785]: SelfCheck: Database status OK.
Nov 30 10:51:29 xxxxx kernel: nf_conntrack: table full, dropping packet.
Nov 30 10:51:43 xxxxx kernel: nf_conntrack: table full, dropping packet.
Nov 30 10:51:47 xxxxx kernel: nf_conntrack: table full, dropping packet.
Nov 30 10:51:54 xxxxx kernel: nf_conntrack: table full, dropping packet.
Nov 30 10:52:00 xxxxx ntpd[3586]: kernel time sync enabled 4001
Nov 30 10:52:04 xxxxx kernel: nf_conntrack: table full, dropping packet.
Nov 30 10:52:18 xxxxx last message repeated 2 times
Nov 30 10:52:26 xxxxx kernel: nf_conntrack: table full, dropping packet.
Nov 30 10:52:38 xxxxx kernel: nf_conntrack: table full, dropping packet.
Nov 30 10:52:58 xxxxx kernel: nf_conntrack: table full, dropping packet.
Nov 30 10:53:23 xxxxx last message repeated 5 times
Nov 30 10:53:30 xxxxx kernel: nf_conntrack: table full, dropping packet.
Nov 30 10:54:06 xxxxx last message repeated 2 times
Nov 30 10:54:28 xxxxx last message repeated 4 times
Nov 30 10:55:24 xxxxx kernel: nf_conntrack: table full, dropping packet.
Nov 30 10:55:25 xxxxx kernel: nf_conntrack: table full, dropping packet.
Nov 30 10:55:44 xxxxx kernel: nf_conntrack: table full, dropping packet.
Nov 30 10:56:12 xxxxx kernel: nf_conntrack: table full, dropping packet.
Nov 30 10:56:46 xxxxx last message repeated 3 times
Nov 30 10:56:55 xxxxx last message repeated 2 times
Nov 30 10:57:28 xxxxx kernel: nf_conntrack: table full, dropping packet.
I've checked online for other solutions - specifically increasing the /proc/sys/net/ipv4/netfilter/ip_conntrack_max
value. It is, however, set to 65536 - its max.

Can anyone offer any advice on where to look to find out what is filling the connection table?


Thanks

Re: nf_conntrack: table full, dropping packet

Posted: Tue Nov 30, 2010 10:58 am
by mikeshinn
Its a safety feature in all Linux kernels, you are tracking too many connections, which can be caused by either misconfigured iptable rules, or just WAY too much traffic. And by way too much, I mean insanely way too much. Its more likely the former, and the later could be caused by a DDOS bounce attack, or something like that (like DNS bounce attacks for example).

Check your firewall rules, and fire up a sniffer and see what kind of traffic is going on. If memory serves, you can set nf_conntrack_max really high, I believe its a 32 bit int, so billions of connections should be possible if you have the RAM for it, but 65K is pretty big, so if you are over that limit check your rules and traffic first - somethings going on there that shouldnt be.

Re: nf_conntrack: table full, dropping packet

Posted: Tue Nov 30, 2010 11:04 am
by chrismcb
Thanks, I'll look at a packet sniffer - in the mean time, here are the IPTables settings:

Code: Select all

 iptables -L
Chain INPUT (policy DROP)
target     prot opt source               destination
DROP       all  --  62.119.28.251        anywhere
DROP       all  --  user-3c2h5u6.cable.mindspring.com  anywhere
DROP       all  --  dsl88-250-50624.ttnet.net.tr  anywhere
DROP       all  --  51-130-178-94.pool.ukrtel.net  anywhere
DROP       all  --  122-36-135-95.pool.ukrtel.net  anywhere
DROP       all  --  88.103.158.23        anywhere
DROP       all  --  host6-133-dynamic.25-79-r.retail.telecomitalia.it  anywhere
DROP       all  --  Static-115.191.96.14.tataidc.co.in  anywhere
DROP       all  --  86.99.114.254        anywhere
DROP       all  --  91.75.74.12          anywhere
DROP       all  --  localhost            anywhere
DROP       all  --  95-174-214-190.nts.su  anywhere
DROP       all  --  121-72-232-248.cable.telstraclear.net  anywhere
DROP       all  --  ABTS-KK-Dynamic-077.141.167.122.airtelbroadband.in  anywhere
DROP       all  --  41.209.75.103        anywhere
DROP       all  --  ep--pc77.static.otenet.gr  anywhere
DROP       all  --  189105032004.user.veloxzone.com.br  anywhere
DROP       all  --  home-pool-164-2.com2com.ru  anywhere
DROP       all  --  195.135.239.5        anywhere
DROP       all  --  109.70.71.60         anywhere
DROP       all  --  144.28.broadband6.iol.cz  anywhere
DROP       all  --  95.67.176.171        anywhere
DROP       all  --  ppp-94-64-145-78.home.otenet.gr  anywhere
DROP       all  --  178.187.137-121.xdsl.ab.ru  anywhere
DROP       all  --  86.35.21.209         anywhere
DROP       all  --  net77.186.188-253.tmn.ertelecom.ru  anywhere
DROP       all  --  250-111-124-91.pool.ukrtel.net  anywhere
DROP       all  --  213.234.13.130       anywhere
DROP       all  --  g43252.upc-g.chello.nl  anywhere
DROP       all  --  41.64.240.72         anywhere
DROP       all  --  adsl190-25105081.dyn.etb.net.co  anywhere
DROP       all  --  sge91-5-88-160-227-197.fbx.proxad.net  anywhere
DROP       all  --  71-33-114-134.spkn.qwest.net  anywhere
DROP       all  --  173-120-215-50.pools.spcsdns.net  anywhere
DROP       all  --  165046.yiuwa.com     anywhere
DROP       all  --  bb171804.virtua.com.br  anywhere
DROP       all  --  ppp95-165-13-236.pppoe.spdop.ru  anywhere
DROP       all  --  186.143.190.167      anywhere
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
REJECT     tcp  --  anywhere             anywhere            tcp flags:!FIN,SYN,RST,ACK/SYN reject-with tcp-reset
DROP       all  --  anywhere             anywhere            state INVALID
ACCEPT     all  --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ksysguard
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:30000
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:pcsync-https
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:cddbp-alt
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ftp
DROP       tcp  --  anywhere             anywhere            tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:submission
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:smtp
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:smtps
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:pop3
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:pop3s
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:imap
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:imaps
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:poppassd
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:mysql
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:postgres
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:9008
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:glrpc
ACCEPT     udp  --  anywhere             anywhere            udp dpt:netbios-ns
ACCEPT     udp  --  anywhere             anywhere            udp dpt:netbios-dgm
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:netbios-ssn
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:microsoft-ds
ACCEPT     udp  --  anywhere             anywhere            udp dpt:openvpn
ACCEPT     udp  --  anywhere             anywhere            udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:domain
ACCEPT     icmp --  anywhere             anywhere            icmp type 8 code 0
ACCEPT     all  --  anywhere             anywhere

Chain FORWARD (policy DROP)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
REJECT     tcp  --  anywhere             anywhere            tcp flags:!FIN,SYN,RST,ACK/SYN reject-with tcp-reset
DROP       all  --  anywhere             anywhere            state INVALID
ACCEPT     all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere

Chain OUTPUT (policy DROP)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
REJECT     tcp  --  anywhere             anywhere            tcp flags:!FIN,SYN,RST,ACK/SYN reject-with tcp-reset
DROP       all  --  anywhere             anywhere            state INVALID
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
Not an expert on this - can anyone have a look over this for me please?

Re: nf_conntrack: table full, dropping packet

Posted: Tue Nov 30, 2010 1:13 pm
by mikeshinn
What are you using your FORWARD rules for?