nf_conntrack: table full, dropping packet

[chrismcb]

Honest answer - I don't know!

These are supposed to be the default rules from the Plesk Firewall module with a port opened for ASL and SSH changed from 22 to 3112 for private access.


By your question, I presume they shouldn't be there?

[biggles]

I am also running the Plesk firewall on one server and it has the same forwarding rules present...

[scott]

Unless you were running as a gateway for something, no those arent needed. Im not sure thats the root of your problem though, unless something crazy were going on (running a proxy for a ddos or something)

[chrismcb]

OK, finally got round to fixing this - rather than just restarting apache.

I turned apache debugging on and got some (many!) core dumps.

The highlights of them all are:

Code: Select all

Core was generated by `/usr/sbin/httpd'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007a3af3961be1 in __libc_res_nquery () from /lib64/libresolv.so.2

Looking at the contents of /proc/net/ip_conntrack, I see thousands of references to my DNS servers.


I've now found the cause, but can anyone offer any advice on a solution?



Thanks

[scott]

Did you ever remove those forwarding rules?

[chrismcb]

Hi Scott,

No, I didnt remove them (until now) as biggles also had them.

I've removed all in the "FORWARD" chain and will keep an eye on the system again.



Thanks

[scott]

The other thing I'd do after this is to remove those nf_conntrack modules. Test it out with the purged FORWARD rules first

[chrismcb]

Thanks - is the nf_conntrack module not needed?

Looking at the IPTABLES list, it shows states in the rules - surely those wouldn't work if the module wasn't loaded?

Code: Select all

ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:11444 state NEW
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:pcsync-https state NEW
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ssh state NEW
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:http state NEW
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:https state NEW
ACCEPT     udp  --  anywhere             anywhere            udp dpt:domain state NEW
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:domain state NEW
ACCEPT     udp  --  anywhere             anywhere            udp dpt:tftp state NEW
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:tftp state NEW
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:smtp state NEW
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:pop3 state NEW
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:imap state NEW
ACCEPT     udp  --  anywhere             anywhere            udp dpt:ntp state NEW
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ftp-data state NEW
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ftp state NEW
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:mysql state NEW
ACCEPT     udp  --  anywhere             anywhere            udp dpt:mysql state NEW
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:cddbp-alt state NEW

[mneese77]

I have those default forwarding rules also when viewed via ssh

Code: Select all

iptables -L -n

, but when looking at my plesk firewall interface, i see no obvious way to remove these forwarding rules....

[chrismcb]

Same with me - I manually removed them from the command line with:

Code: Select all

iptables -F FORWARD

Then restarted iptables.

Not saved it to the system yet, but will do if it shows an improvement.


I still don't believe that this is the cause of the problem, but every little helps!


EDIT: iptables switch is -F for flush rather than -X which I had before.

[mneese77]

If you removed them with that simple command, then re-started the table, doesn't that save them to the system? Or is there another step to this process?

[chrismcb]

I believe it will save for the current session, but if you want to save permenantly, you must use:

Code: Select all

iptables-save

If you don't, your previously saved rules will be reapplied on restart.

[chrismcb]

Just had the same issue again - with the forward rules removed.

Anyone able to offer advice as to the error message from the apache crash dump?

Code: Select all

Core was generated by `/usr/sbin/httpd'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007a3af3961be1 in __libc_res_nquery () from /lib64/libresolv.so.2

The file is provided by glibc and the currently installed version (latest) is:

Code: Select all

glibc-2.5-49.el5_5.7

[chrismcb]

Anyone able to help?

I'm now finding myself having to use my mobile to SSH into the server when this happens and restart apache to clear the connections - while hoping none of my clients catch on that the CPU is being thrashed and websites/emails are temporarily down.

[chrismcb]

I've tried to go further with the apache dumps that I gathered and hoping to get down to the function call which caused the problem.

I'm using gdb and have attached the dump and trace.


Following sources online (specifically http://www.omh.cc/blog/2008/mar/6/fixin ... aused-php/), I have been trying to use the following command:

Code: Select all

dump_bt executor_globals.current_execute_data

This should return a nicer backtrace - however, I'm getting:

Code: Select all

Attempt to extract a component of a value that is not a structure.

Any help is appreciated...