nf_conntrack: table full, dropping packet

General Discussion of atomic repo and development projects.

Ask for help here with anything else not covered by other forums.
chrismcb
Forum Regular
Forum Regular
Posts: 293
Joined: Tue Nov 23, 2010 7:30 am
Location: Glasgow, UK

nf_conntrack: table full, dropping packet

Unread post by chrismcb »

Hi,

I installed ASL yesterday after a successful hack on my server - I've went through all the steps to get it installed and now have 1 moderate and 1 low vulnerability.

Everything was operating fine until a few minute ago when the server completely stopped responding.

I could not log in via SSH, view websites, use emails or even ping.


I used my remote console to reset the power and reboot.


After inspecting /var/log/messages, i can see that the server was doing things while it was unresponsive:

Code: Select all

Nov 24 18:27:43 xxx xinetd[3079]: EXIT: ftp status=0 pid=16002 duration=52(sec)
Nov 24 18:28:59 xxx clamd[19629]: stream(127.0.0.1@1027): ASL.MalwareBlacklist.rbcmail.ru.UNOFFICIAL FOUND
Nov 24 18:29:08 xxx kernel: nf_conntrack: table full, dropping packet.
Nov 24 18:30:01 xxx psmon[16118]: Forking background daemon, process 16119.
Nov 24 18:30:01 xxx psmon[16119]: Forking second background daemon, process 16121.
Nov 24 18:31:00 xxx kernel: nf_conntrack: table full, dropping packet.
Nov 24 18:31:07 xxx postfix/smtpd[16131]: sql_sqlite3 plugin: no result found
Nov 24 18:32:13 xxx kernel: nf_conntrack: table full, dropping packet.
Nov 24 18:33:21 xxx syslogd 1.4.1: restart.
I'm assuming here that clamd found a virus in an email - nothing unsual about that (or am I wrong?).

What concerns me is this new log entry that i've never seen before:

Code: Select all

kernel: nf_conntrack: table full, dropping packet.
Can anyone advise on what it is doing?


Thanks


p.s. The sql_sqlite log is one i've seen for months - I tried to get rid of it, but couldn't figure out how
Last edited by chrismcb on Wed Nov 24, 2010 3:24 pm, edited 1 time in total.
chrismcb
Forum Regular
Forum Regular
Posts: 293
Joined: Tue Nov 23, 2010 7:30 am
Location: Glasgow, UK

Re: nf_conntrack: table full, dropping packet

Unread post by chrismcb »

OK, so after the restart, all websites went down amost immediately again.

The log shows:

Code: Select all

Nov 24 18:34:51 xxx kernel: PAX: From 217.44.79.193: execution attempt in: <anonymous mapping>, 9e6556af000-9e6615cb000 9e6556af000
Nov 24 18:34:51 xxx kernel: PAX: terminating task: /usr/sbin/httpd(httpd):4076, uid/euid: 48/48, PC: 000009e660cc9c30, SP: 00007fffdee8e5f8
Nov 24 18:34:51 xxx kernel: PAX: bytes at PC: 01 00 00 00 e6 09 00 00 20 9c cc 60 e6 09 00 00 60 01 00 00
Nov 24 18:34:51 xxx kernel: PAX: bytes at SP-8: 0000627cb0711ac9 0000627cb0629d1d 000009e65c61e3f0 00007fffdee90af8 0000000000000000 000009e660cdad48 0000000000000000 000000005f562f70 000009e65f2416a0 000009e65cce9c60 0000000000000000
Nov 24 18:34:51 xxx kernel: PAX: From 67.195.115.185: execution attempt in: <anonymous mapping>, 9e6556af000-9e6613c2000 9e6556af000
Nov 24 18:34:51 xxx kernel: PAX: terminating task: /usr/sbin/httpd(httpd):3816, uid/euid: 48/48, PC: 000009e661091b20, SP: 00007fffdee91078
Nov 24 18:34:51 xxx kernel: PAX: bytes at PC: 01 00 00 00 e6 09 00 00 10 1b 09 61 e6 09 00 00 60 01 00 00
Nov 24 18:34:51 xxx kernel: PAX: bytes at SP-8: 0000627cb0711ac9 0000627cb0629d1d 000009e65c61e3f0 00007fffdee92458 0000000000000000 000009e660ac0510 0000000000000000 000000005f562f70 000009e65f2416a0 000009e65cce9c60 0000000000000000
I restarted httpd manually and it's working now.

Are these related? Could the server still be under attack?
Last edited by chrismcb on Wed Nov 24, 2010 3:23 pm, edited 1 time in total.
BruceLee
Forum Regular
Forum Regular
Posts: 879
Joined: Sat Mar 28, 2009 6:58 pm
Location: Germany

Re: nf_conntrack: table full, dropping packet

Unread post by BruceLee »

One very important notice.
ASL is mainly focused on preventing attacks and securing your server with a lot of useful addtions. Of course it scans and detects a lot of bad things but it's not intended to be installed for cleaning an almost hacked server. You should definitely research your problems and fix them. You can contact atomicorp to get a quote for that.

Concerning your PAX log entries:
http://www.atomicorp.com/wiki/index.php ... st_mean.3F
chrismcb
Forum Regular
Forum Regular
Posts: 293
Joined: Tue Nov 23, 2010 7:30 am
Location: Glasgow, UK

Re: nf_conntrack: table full, dropping packet

Unread post by chrismcb »

Thanks - I will contact them for a quote just now.

I am, in the mean time, trying to get to the bottom of the unresponsiveness that forced me to reboot.

After the hack, everything was restored and ASL installed immediately.

I do realise that whatever vulnerability allowed the hack to tak place is possibly still there.


Thanks for the link about the PAX messages - I figured out what was producing the message, but it was the actual message and its actions which concerned me:

Code: Select all

Nov 24 18:34:51 xxx kernel: PAX: From 217.44.79.193: execution attempt in: <anonymous mapping>, 9e6556af000-9e6615cb000 9e6556af000
Nov 24 18:34:51 xxx kernel: PAX: terminating task: /usr/sbin/httpd(httpd):4076, uid/euid: 48/48, PC: 000009e660cc9c30, SP: 00007fffdee8e5f8
To me that looks like some remote computer has tried to execute something and ASL has shut httpd down to protect itself?
Is that right?
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: nf_conntrack: table full, dropping packet

Unread post by mikeshinn »

Exactly, someone tried to exploit a vulnerability in apache and ASL stopped apache from compromising your server - so if I understand correctly (and please correct me if I am wrong) was your systems compromised previously? If so then you really should reinstall the system from a known trusted image, there is no tool that can guarantee that its removed all the malware, rootkits, etc. from your system - the best (and quickest) solution is to reinstall the box, then lock it down.
chrismcb
Forum Regular
Forum Regular
Posts: 293
Joined: Tue Nov 23, 2010 7:30 am
Location: Glasgow, UK

Re: nf_conntrack: table full, dropping packet

Unread post by chrismcb »

Yes, the server was compromised yesterday - however I cannot afford a complete reinstall as I have clients currently using the server with no current alternative.

The two IP adresses in the logs were for a computer local to my location probably looking at a website for one of my clients and the Yahoo crawl bot.

I wouldn't have expected these to cause ASL to shut apache down.

Can I find out somehow what it was that caused apache to be shut down?
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: nf_conntrack: table full, dropping packet

Unread post by mikeshinn »

Thats hard to say, if the system is compromised it could be anything, you could have a compromised library, apache itself, some module, even a script - the box is basically in a corrupted state. Someone else had root on your box and installed software on it, designed to hide itself, so the root cause of apache dying could really be anything. There are, unfortunately, no easy answers in this case. Its like your body getting possessed, no telling what it might do and no telling why either.
chrismcb
Forum Regular
Forum Regular
Posts: 293
Joined: Tue Nov 23, 2010 7:30 am
Location: Glasgow, UK

Re: nf_conntrack: table full, dropping packet

Unread post by chrismcb »

OK, I'm biting the bullet and purchasing a new server from my hosting company to migrate all websites/emails to using the Plesk migration tool.

What advice can you give to setting up a new server?

Update the system, install ASL and follow its guidance to secure things?
Anything else?


Your help is much appreciated.
BruceLee
Forum Regular
Forum Regular
Posts: 879
Joined: Sat Mar 28, 2009 6:58 pm
Location: Germany

Re: nf_conntrack: table full, dropping packet

Unread post by BruceLee »

Additionaly I would setup a firewall and configure it as secure as you can. afp or the plesk one.
If your provider provides a firewall in front of your server I would configure that one too.
Besides that a backup system/script/whatever you prefer is mandatory.
An antivirus solution is mandatory too. like clamav from atomicorp.
spamassassin, rbl and greylisting. rkhunter. some kind of monitoring for cpu, processes, diskspace etc.

I bet others will have some extra tipps I haven't thought of or forgot them :)
Kalimari
Forum Regular
Forum Regular
Posts: 526
Joined: Wed Jan 02, 2008 3:21 pm
Location: United Kingdom

Re: nf_conntrack: table full, dropping packet

Unread post by Kalimari »

All excellent tip's from BruceLee!

As you're not sure how the bad guys got into your server, disable direct root SSH log-in and add a no privileges account which can su - to root, ensure you use NEW SSH PASSWORDS or combine this approach with certificate authentication. ASL will highlight the latter as a priority (if you've not done so already, its worth reading up before hand, there are several guides on-line covering Securing SSH with Key Based Authentication).

If you have a static IP and its feasible, only allow that addresses to connect via SSH (using Plesk/AFP firewall).

Before migrating website, run: rkhunter -c -sk & clamscan -r -i /var/www/vhosts/ - both part of the atomicorp/ASL install to ensure you do not transfer badness!

Opting for the new server is the only realistic option, the inconvenience of migrating sites is minor compared to being owned over and over again.

Good luck!
chrismcb
Forum Regular
Forum Regular
Posts: 293
Joined: Tue Nov 23, 2010 7:30 am
Location: Glasgow, UK

Re: nf_conntrack: table full, dropping packet

Unread post by chrismcb »

Excellent advice guys.

Once all is put down on paper (or is it on screen?) its so obvoius - will get those done asap.


Had problems upgrading Plesk - but got them sorted now.

Will get the new server configured for a (hopefully) smooth transfer.
BruceLee
Forum Regular
Forum Regular
Posts: 879
Joined: Sat Mar 28, 2009 6:58 pm
Location: Germany

Re: nf_conntrack: table full, dropping packet

Unread post by BruceLee »

concering the security: I set all the passwords for the users at the beginning and give them the tool keepass (keepass.info) with their db. I advice them to use it and let it generate passwords.
So take strong ones for everything. Of course it doens't help if a client computer got hijacked and the password is sniffed but it prevent simple brute or guess attacks.

Your Plesk problem is that Plesk 9 has a bug where every update/installation replaces postfix with qmail and vice versa.
If you want to use qmail exclude postfix package in plesk.repo file via exclude=psa-mail-pc-driver*
If you want to use postfix exclude qmail package in plesk.repo file via exclude=psa-mail-qc-driver*
chrismcb
Forum Regular
Forum Regular
Posts: 293
Joined: Tue Nov 23, 2010 7:30 am
Location: Glasgow, UK

Re: nf_conntrack: table full, dropping packet

Unread post by chrismcb »

Yes, I'm learning that now about passwords.

Any ones brought to my attention with ASL will have their passwords updated - and i'll set passwords up for them in the future, as you suggest.


Everything seems configured and happy now.

Just going to add the Atomic repo for one final update then i'll sit tight until close of business.
BruceLee
Forum Regular
Forum Regular
Posts: 879
Joined: Sat Mar 28, 2009 6:58 pm
Location: Germany

Re: nf_conntrack: table full, dropping packet

Unread post by BruceLee »

if you update with atomic repo than mysql will be updated.
take a look in the wiki how to do it:
http://www.atomicorp.com/wiki/index.php/Mysql
chrismcb
Forum Regular
Forum Regular
Posts: 293
Joined: Tue Nov 23, 2010 7:30 am
Location: Glasgow, UK

Re: nf_conntrack: table full, dropping packet

Unread post by chrismcb »

Thanks - i've used the atomic repo in the past for updates.

Having dependancy issues again though - this time with sqlite2... the investigation continues!
Post Reply