nf_conntrack: table full, dropping packet

[scott]

Did you ever remove those FORWARD rules?

[chrismcb]

Just had the same issue again - with the forward rules removed.

Yep - took them out, still the same.


It's definately something to do with outgoing connections using DNS - when it happens, I can view the contents of /proc/net/ip_conntrack and it lists connections with destination IP addresses of my DNS server addresses in /etc/resolv.conf

[scott]

and did you remove all the connection tracking modules?

[mikeshinn]

Are you looping your DNS traffic back to yourself through your eternal IP address?

[chrismcb]

@Scott - No, I never removed the module as the firewall rules by Plesk look as if they rely on state being recorded.

If I removed the module which tracks connection state, would the firewall not then block all traffic?


@Mike - sounds like a plausible problem - how would I know and what should I check?

/etc/resolv.conf:

Code: Select all

nameserver 208.67.220.220
nameserver 208.67.222.222

[mikeshinn]

If those are your only resolver entries you're just sending your DNS requests to openDNS (good folks BTW) and not looping to yourself - but what does your tracking table look like?

You may actually just have so many DNS queries open that you're filling up your tracking table. Did I mention we wrote a really inexpensive book on this topic that would do wonders to help you troubleshoot your firewall?

http://www.amazon.com/Troubleshooting-L ... 065&sr=8-1

Hell, there are used copies on amazon for under $5.

[chrismcb]

Yep - I like OpenDNS - saves headaches when dealing with propagation.

Anyway... The entry table is fine just now - it varies from a few hundred to a max (that i've noticed) of around 1500.

When this problem occurs, it maxes the table out at 65535.


When I view the table, it is full of requests to the OpenDNS servers.

I know that's what's causing the table to become full, but unsure as to how to find out what's causing it to fill.


The apache dump has led me to libresolv.so.2

Code: Select all

Core was generated by `/usr/sbin/httpd'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007a3af3961be1 in __libc_res_nquery () from /lib64/libresolv.so.2

Either it is at fault (which I doubt) or something is repeatedly calling it?

The question is now, how to find out what's calling it?


From my post on Fri Dec 10, 2010 5:12 pm - I have tried to use GDB to find out what php function is last called, but ended up with the error explained.


Hopefully someone can help me - even just confirm I'm on the right tracks?



Thanks

[scott]

Hmm, well any of the web statistics apps would generate a high volume of queries. Mail could also do that, depending on the volume.

[chrismcb]

Thanks,

I don't have any stats packages installed - other than the standard ones from Plesk.

This has only started since either 1) installing ASL or 2) some other web app that I have created.


Can you offer advice on any monitoring program I can install or specific tasks I can run to try to pinpoint the source of all of the traffic?

[BruceLee]

you could use iftop to check which connection uses lot bandwidth and wireshark for sniffing.
basically it still could be a comprimised system through a vhost since moving to a new server with the same vhosts credentials (that were allready "hacked/used" by the bad guys.) does not guarantee anything.
Or some sort of misconfiguration.

[chrismcb]

Great, thanks.

I've installed both iftop and Wireshark on the server - and the Wireshark GUI on my computer for analysis after capture.


Never used Wireshark before are there any useful switches you can reccomend to enhance (or strip out superflous info from) the capture?


I'll now have to wait for it to happen again and hope that I'm sitting at my desk ready for it!

[mikeshinn]

On a server you really should run your own resolver, and if you want to do you requests to openDNS just configure bind to use them as your forwarder. Configuring a server to use an upstream DNS server is going to kill your performance.

[chrismcb]

Mike - I've set the nameservers back to my hosting company's internal name servers as the server was inially configured for - this will reduce overheads.

I can enable OpenDNS again, if required, in the future.


Didn't take too long before the error occurred again.
With a 45seconds of capture from Wireshark, over 500MB of capture data was saved.

Analysing it with my local GUI installation on Windows of Wireshark, almost all are originating from a shared IP address on my server with a destination address of 41.225.9.6 - an IP address in Tunisia.


Typical output from Wireshark is:



Never having used Wireshark, can anyone offer advice to dig deeper to see what is being sent to this foreign IP address to try to trace it back to a source file?

[chrismcb]

I changed some of the options for the apache crash dumps I have been collecting - and they were not appearing in the drectory i set it to.

I subsequently found them in a directory in a vhost.


As BruceLee pointed out, comprmmised data from a vhost could have been transferred.

Looks like you were right!

I found some strange scripts in the directory the dumps were residing.



All removed now - hopefully that was the source of the problems!

Thanks BruceLee, would never have thought to go through vhosts!

[BruceLee]

Glad you could pinpoint it. Now you still need to change the ftp and plesk user credentials of the vhost so that the bad guys dont upload anything again.