Page 1 of 5

nf_conntrack: table full, dropping packet

Posted: Wed Nov 24, 2010 2:55 pm
by chrismcb
Hi,

I installed ASL yesterday after a successful hack on my server - I've went through all the steps to get it installed and now have 1 moderate and 1 low vulnerability.

Everything was operating fine until a few minute ago when the server completely stopped responding.

I could not log in via SSH, view websites, use emails or even ping.


I used my remote console to reset the power and reboot.


After inspecting /var/log/messages, i can see that the server was doing things while it was unresponsive:

Code: Select all

Nov 24 18:27:43 xxx xinetd[3079]: EXIT: ftp status=0 pid=16002 duration=52(sec)
Nov 24 18:28:59 xxx clamd[19629]: stream(127.0.0.1@1027): ASL.MalwareBlacklist.rbcmail.ru.UNOFFICIAL FOUND
Nov 24 18:29:08 xxx kernel: nf_conntrack: table full, dropping packet.
Nov 24 18:30:01 xxx psmon[16118]: Forking background daemon, process 16119.
Nov 24 18:30:01 xxx psmon[16119]: Forking second background daemon, process 16121.
Nov 24 18:31:00 xxx kernel: nf_conntrack: table full, dropping packet.
Nov 24 18:31:07 xxx postfix/smtpd[16131]: sql_sqlite3 plugin: no result found
Nov 24 18:32:13 xxx kernel: nf_conntrack: table full, dropping packet.
Nov 24 18:33:21 xxx syslogd 1.4.1: restart.
I'm assuming here that clamd found a virus in an email - nothing unsual about that (or am I wrong?).

What concerns me is this new log entry that i've never seen before:

Code: Select all

kernel: nf_conntrack: table full, dropping packet.
Can anyone advise on what it is doing?


Thanks


p.s. The sql_sqlite log is one i've seen for months - I tried to get rid of it, but couldn't figure out how

Re: nf_conntrack: table full, dropping packet

Posted: Wed Nov 24, 2010 2:58 pm
by chrismcb
OK, so after the restart, all websites went down amost immediately again.

The log shows:

Code: Select all

Nov 24 18:34:51 xxx kernel: PAX: From 217.44.79.193: execution attempt in: <anonymous mapping>, 9e6556af000-9e6615cb000 9e6556af000
Nov 24 18:34:51 xxx kernel: PAX: terminating task: /usr/sbin/httpd(httpd):4076, uid/euid: 48/48, PC: 000009e660cc9c30, SP: 00007fffdee8e5f8
Nov 24 18:34:51 xxx kernel: PAX: bytes at PC: 01 00 00 00 e6 09 00 00 20 9c cc 60 e6 09 00 00 60 01 00 00
Nov 24 18:34:51 xxx kernel: PAX: bytes at SP-8: 0000627cb0711ac9 0000627cb0629d1d 000009e65c61e3f0 00007fffdee90af8 0000000000000000 000009e660cdad48 0000000000000000 000000005f562f70 000009e65f2416a0 000009e65cce9c60 0000000000000000
Nov 24 18:34:51 xxx kernel: PAX: From 67.195.115.185: execution attempt in: <anonymous mapping>, 9e6556af000-9e6613c2000 9e6556af000
Nov 24 18:34:51 xxx kernel: PAX: terminating task: /usr/sbin/httpd(httpd):3816, uid/euid: 48/48, PC: 000009e661091b20, SP: 00007fffdee91078
Nov 24 18:34:51 xxx kernel: PAX: bytes at PC: 01 00 00 00 e6 09 00 00 10 1b 09 61 e6 09 00 00 60 01 00 00
Nov 24 18:34:51 xxx kernel: PAX: bytes at SP-8: 0000627cb0711ac9 0000627cb0629d1d 000009e65c61e3f0 00007fffdee92458 0000000000000000 000009e660ac0510 0000000000000000 000000005f562f70 000009e65f2416a0 000009e65cce9c60 0000000000000000
I restarted httpd manually and it's working now.

Are these related? Could the server still be under attack?

Re: nf_conntrack: table full, dropping packet

Posted: Wed Nov 24, 2010 3:17 pm
by BruceLee
One very important notice.
ASL is mainly focused on preventing attacks and securing your server with a lot of useful addtions. Of course it scans and detects a lot of bad things but it's not intended to be installed for cleaning an almost hacked server. You should definitely research your problems and fix them. You can contact atomicorp to get a quote for that.

Concerning your PAX log entries:
http://www.atomicorp.com/wiki/index.php ... st_mean.3F

Re: nf_conntrack: table full, dropping packet

Posted: Wed Nov 24, 2010 3:23 pm
by chrismcb
Thanks - I will contact them for a quote just now.

I am, in the mean time, trying to get to the bottom of the unresponsiveness that forced me to reboot.

After the hack, everything was restored and ASL installed immediately.

I do realise that whatever vulnerability allowed the hack to tak place is possibly still there.


Thanks for the link about the PAX messages - I figured out what was producing the message, but it was the actual message and its actions which concerned me:

Code: Select all

Nov 24 18:34:51 xxx kernel: PAX: From 217.44.79.193: execution attempt in: <anonymous mapping>, 9e6556af000-9e6615cb000 9e6556af000
Nov 24 18:34:51 xxx kernel: PAX: terminating task: /usr/sbin/httpd(httpd):4076, uid/euid: 48/48, PC: 000009e660cc9c30, SP: 00007fffdee8e5f8
To me that looks like some remote computer has tried to execute something and ASL has shut httpd down to protect itself?
Is that right?

Re: nf_conntrack: table full, dropping packet

Posted: Wed Nov 24, 2010 4:44 pm
by mikeshinn
Exactly, someone tried to exploit a vulnerability in apache and ASL stopped apache from compromising your server - so if I understand correctly (and please correct me if I am wrong) was your systems compromised previously? If so then you really should reinstall the system from a known trusted image, there is no tool that can guarantee that its removed all the malware, rootkits, etc. from your system - the best (and quickest) solution is to reinstall the box, then lock it down.

Re: nf_conntrack: table full, dropping packet

Posted: Wed Nov 24, 2010 4:54 pm
by chrismcb
Yes, the server was compromised yesterday - however I cannot afford a complete reinstall as I have clients currently using the server with no current alternative.

The two IP adresses in the logs were for a computer local to my location probably looking at a website for one of my clients and the Yahoo crawl bot.

I wouldn't have expected these to cause ASL to shut apache down.

Can I find out somehow what it was that caused apache to be shut down?

Re: nf_conntrack: table full, dropping packet

Posted: Wed Nov 24, 2010 5:21 pm
by mikeshinn
Thats hard to say, if the system is compromised it could be anything, you could have a compromised library, apache itself, some module, even a script - the box is basically in a corrupted state. Someone else had root on your box and installed software on it, designed to hide itself, so the root cause of apache dying could really be anything. There are, unfortunately, no easy answers in this case. Its like your body getting possessed, no telling what it might do and no telling why either.

Re: nf_conntrack: table full, dropping packet

Posted: Thu Nov 25, 2010 5:35 am
by chrismcb
OK, I'm biting the bullet and purchasing a new server from my hosting company to migrate all websites/emails to using the Plesk migration tool.

What advice can you give to setting up a new server?

Update the system, install ASL and follow its guidance to secure things?
Anything else?


Your help is much appreciated.

Re: nf_conntrack: table full, dropping packet

Posted: Thu Nov 25, 2010 6:54 am
by BruceLee
Additionaly I would setup a firewall and configure it as secure as you can. afp or the plesk one.
If your provider provides a firewall in front of your server I would configure that one too.
Besides that a backup system/script/whatever you prefer is mandatory.
An antivirus solution is mandatory too. like clamav from atomicorp.
spamassassin, rbl and greylisting. rkhunter. some kind of monitoring for cpu, processes, diskspace etc.

I bet others will have some extra tipps I haven't thought of or forgot them :)

Re: nf_conntrack: table full, dropping packet

Posted: Thu Nov 25, 2010 8:22 am
by Kalimari
All excellent tip's from BruceLee!

As you're not sure how the bad guys got into your server, disable direct root SSH log-in and add a no privileges account which can su - to root, ensure you use NEW SSH PASSWORDS or combine this approach with certificate authentication. ASL will highlight the latter as a priority (if you've not done so already, its worth reading up before hand, there are several guides on-line covering Securing SSH with Key Based Authentication).

If you have a static IP and its feasible, only allow that addresses to connect via SSH (using Plesk/AFP firewall).

Before migrating website, run: rkhunter -c -sk & clamscan -r -i /var/www/vhosts/ - both part of the atomicorp/ASL install to ensure you do not transfer badness!

Opting for the new server is the only realistic option, the inconvenience of migrating sites is minor compared to being owned over and over again.

Good luck!

Re: nf_conntrack: table full, dropping packet

Posted: Thu Nov 25, 2010 8:27 am
by chrismcb
Excellent advice guys.

Once all is put down on paper (or is it on screen?) its so obvoius - will get those done asap.


Had problems upgrading Plesk - but got them sorted now.

Will get the new server configured for a (hopefully) smooth transfer.

Re: nf_conntrack: table full, dropping packet

Posted: Thu Nov 25, 2010 9:25 am
by BruceLee
concering the security: I set all the passwords for the users at the beginning and give them the tool keepass (keepass.info) with their db. I advice them to use it and let it generate passwords.
So take strong ones for everything. Of course it doens't help if a client computer got hijacked and the password is sniffed but it prevent simple brute or guess attacks.

Your Plesk problem is that Plesk 9 has a bug where every update/installation replaces postfix with qmail and vice versa.
If you want to use qmail exclude postfix package in plesk.repo file via exclude=psa-mail-pc-driver*
If you want to use postfix exclude qmail package in plesk.repo file via exclude=psa-mail-qc-driver*

Re: nf_conntrack: table full, dropping packet

Posted: Thu Nov 25, 2010 10:01 am
by chrismcb
Yes, I'm learning that now about passwords.

Any ones brought to my attention with ASL will have their passwords updated - and i'll set passwords up for them in the future, as you suggest.


Everything seems configured and happy now.

Just going to add the Atomic repo for one final update then i'll sit tight until close of business.

Re: nf_conntrack: table full, dropping packet

Posted: Thu Nov 25, 2010 10:06 am
by BruceLee
if you update with atomic repo than mysql will be updated.
take a look in the wiki how to do it:
http://www.atomicorp.com/wiki/index.php/Mysql

Re: nf_conntrack: table full, dropping packet

Posted: Thu Nov 25, 2010 10:10 am
by chrismcb
Thanks - i've used the atomic repo in the past for updates.

Having dependancy issues again though - this time with sqlite2... the investigation continues!