IP whitelist increases the server load

[scott]

There you go then. Fast firewall code is in 3.2.64+ btw.

[octet]

OK, made this the default kernel, need to find a good time for a reboot I'll report back.

title CentOS (3.2.64-75.art.x86_64)

[octet]

[root@alien ~]# uname -a
Linux alien.3dev.biz 3.2.64-75.art.x86_64 #1 SMP Thu Dec 4 13:57:49 EST 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@alien ~]#

Let's hope we won't have any hiccups, I will report back in few days if it's load free

Thanks Scott!

[iv@rh]

Ah so its iptables on your system huh. Thats completely different from what we're seeing on one of the other systems in this threat, there its mysql related.

Are you using the ASL kernel? This would let you use the faster firewall system (it does not use iptables at all), and do you have large geo or local blacklists?

Scott.

There is no iptables running, I just added it for comparison of the effectiveness.

Code: Select all

chkconfig --list iptables
iptables       	0:off	1:off	2:off	3:off	4:off	5:off	6:off

Also, as I mentioned, along with the original poster, we use Cloudlinux (and it's kernel). No geo blacklist at all, local blacklist has 15 IP addresses, blocklist has 88 IP addresses.

This problem also when white/black-listing via the ASL console.

This problem is so frustrating, defeats the purpose of using ASL (imaging how happy our customers are by having their web sites secured, but down for at least 30 minutes every day). Glad more people are posting here, as this highlights the problem is not local to 2 servers.

Is there a solution Atomicorp could suggest?

[iv@rh]

Ok, we use kernel

Code: Select all

uname -a
Linux [hostname] 2.6.32-531.29.2.lve1.3.11.el6.x86_64 #1 SMP Tue Nov 25 02:38:02 EST 2014 x86_64 x86_64 x86_64 GNU/Linux

Code: Select all

cat /etc/redhat-release
CloudLinux Server release 6.6 (Leonid Kizim)

So you are saying firewall is unusable in 2.6.xx kernels? Does this mean ASL is not compatible with 2.6.xx kernel any more? It was working just fine until a few months ago.

Just more update, there is kswapd running in top, indicating file transfer. I was unable to use iotop or sysdig because they do not respond within 10 minutes (the server is that busy, they do not even start). When start sysdig in advance, it stops refreshing when the problem occurs.

Also other poster's comment about mysqld sitting in the top is only indication of heavy loaded server doing heavy IO, and mysql does IO at the same time.
tuning mysql buffer_pool_size can improve it but meaningless to adjust to deal with this problem.

[scott]

2.6.32's netfilter is somewhat dated, in ASL if we do not detect the newer system it will automatically fall back on the older iptables interface which isnt as fast as the newer components. For example a large geo-blacklist (say 200k entries) takes several minutes if not longer on even an idle system, resulting in both high memory overhead and high disk IO. Additionally this method results in a gap period where the firewall policy is esentially open while the policy is loading.

The newer method (using sets and hashes) can not only load the entire geo-db (300k entries) in a fraction of a millisecond, it also lets us swap sets. So you can have a new set loading on an update, and swap places with it without any kind of open state in the firewal policy at all. Its considerably more advanced, and when you get into extremely large firewall policies its really the only way to go.

So could it be the firewall policy? Possibly, and I know for a fact that mysql can be be blocking here as well. It could be a combination, or something else entirely. Until we get a chance to look at it I'm speculating here. We do have one case open to look at a system reporting this, so we'll probably start there. If you don't have a case open already, go ahead and open one up if its something we can look at first hand.

[iv@rh]

Thank you for explaining Scott. I have a few valid concerns, though:

1. cPanel/CentOS 6.x are all running 2.6.32.x kernels. So if ASL claims it is compatible with cPanel, it must support these kernels efficiently. Even if you add exceptions in your scripts to reduce some of the logic and avoid entering high IO state.
2. ASL was working fine until a few months ago. I confirm we were extensively using asl -bl [IP] and asl -wl [IP] before without any issues.

And lastly, I am unable to access the support portal, as your password reset link resets everything else but the login password. This is the reason I am searching your forums for help.

[hostingg]

1. cPanel/CentOS 6.x are all running 2.6.32.x kernels.

No they arent, im running the 3.2.62-74 ASL kernel with Centos and Cpanel. Cpanel doesnt include a kernel or care what kernel you use.

For what its worth, im not seeing any issues with the newer kernels.

[scott]

Believe me if we could make 2.6.18 or 2.6.32's firewall more efficient we would, but there is honestly nothing more that can be done here.

You can email your support requests to support@atomicorp.com too, and bring up that you cant log into the portal there.

[mikeshinn]

Is there any reason folks arent using the new kernels? 2.6.32 is ancient, and 2.6.18 is almost in the stone age compared to the improvements in 3.x.

[copernic2006]

Is there any reason folks arent using the new kernels? 2.6.32 is ancient, and 2.6.18 is almost in the stone age compared to the improvements in 3.x.

In our case, we use CloudLinux (Kernel: 2.6.32-531.29.2.lve1.3.11.1.el6.x86_64)
It is impossible for us to do without CloudLinux especially for its resource management functionality per account (including mysql-governor).

[iv@rh]

Is there any reason folks arent using the new kernels? 2.6.32 is ancient, and 2.6.18 is almost in the stone age compared to the improvements in 3.x.

In our case, we use CloudLinux (Kernel: 2.6.32-531.29.2.lve1.3.11.1.el6.x86_64)
It is impossible for us to do without CloudLinux especially for its resource management functionality per account (including mysql-governor).

Same here. Cloudlinux wins in terms of performance for the shared hosting, as some customers abuse resources of the whole server.

[iv@rh]

It worth mentioning that your 3.x kernels do not work in xenserver environments. VM simply not booting after changing the kernel to 3.2.x (at least for CentOS 6.x)

We have just upgraded to the latest XenServer 6.5 which is based on 3.2.x kernel for Dom0 and has improved support for 3.x kernels in DomU, but still no luck with your kernels.

[scott]

There is a xen specific kernel channel you need to use there, called tortix-kernel-xen. This is what we use on AWS and Rackspace.

[hostingg]

CloudLinux (Kernel: 2.6.32-531.29.2.lve1.3.11.1.el6.x86_64)

why do they use such an old kernel? looking onkernel.org its no longer supported or maintained.