IP whitelist increases the server load
-
- Atomicorp Staff - Site Admin
- Posts: 8355
- Joined: Wed Dec 31, 1969 8:00 pm
- Location: earth
- Contact:
Re: IP whitelist increases the server load
There you go then. Fast firewall code is in 3.2.64+ btw.
Re: IP whitelist increases the server load
OK, made this the default kernel, need to find a good time for a reboot I'll report back.
title CentOS (3.2.64-75.art.x86_64)
title CentOS (3.2.64-75.art.x86_64)
Re: IP whitelist increases the server load
[root@alien ~]# uname -a
Linux alien.3dev.biz 3.2.64-75.art.x86_64 #1 SMP Thu Dec 4 13:57:49 EST 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@alien ~]#
Let's hope we won't have any hiccups, I will report back in few days if it's load free
Thanks Scott!
Linux alien.3dev.biz 3.2.64-75.art.x86_64 #1 SMP Thu Dec 4 13:57:49 EST 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@alien ~]#
Let's hope we won't have any hiccups, I will report back in few days if it's load free
Thanks Scott!
Re: IP whitelist increases the server load
Scott.scott wrote:Ah so its iptables on your system huh. Thats completely different from what we're seeing on one of the other systems in this threat, there its mysql related.
Are you using the ASL kernel? This would let you use the faster firewall system (it does not use iptables at all), and do you have large geo or local blacklists?
There is no iptables running, I just added it for comparison of the effectiveness.
Code: Select all
chkconfig --list iptables
iptables 0:off 1:off 2:off 3:off 4:off 5:off 6:off
This problem also when white/black-listing via the ASL console.
This problem is so frustrating, defeats the purpose of using ASL (imaging how happy our customers are by having their web sites secured, but down for at least 30 minutes every day). Glad more people are posting here, as this highlights the problem is not local to 2 servers.
Is there a solution Atomicorp could suggest?
Re: IP whitelist increases the server load
Ok, we use kernel
So you are saying firewall is unusable in 2.6.xx kernels? Does this mean ASL is not compatible with 2.6.xx kernel any more? It was working just fine until a few months ago.
Just more update, there is kswapd running in top, indicating file transfer. I was unable to use iotop or sysdig because they do not respond within 10 minutes (the server is that busy, they do not even start). When start sysdig in advance, it stops refreshing when the problem occurs.
Also other poster's comment about mysqld sitting in the top is only indication of heavy loaded server doing heavy IO, and mysql does IO at the same time.
tuning mysql buffer_pool_size can improve it but meaningless to adjust to deal with this problem.
Code: Select all
uname -a
Linux [hostname] 2.6.32-531.29.2.lve1.3.11.el6.x86_64 #1 SMP Tue Nov 25 02:38:02 EST 2014 x86_64 x86_64 x86_64 GNU/Linux
Code: Select all
cat /etc/redhat-release
CloudLinux Server release 6.6 (Leonid Kizim)
Just more update, there is kswapd running in top, indicating file transfer. I was unable to use iotop or sysdig because they do not respond within 10 minutes (the server is that busy, they do not even start). When start sysdig in advance, it stops refreshing when the problem occurs.
Also other poster's comment about mysqld sitting in the top is only indication of heavy loaded server doing heavy IO, and mysql does IO at the same time.
tuning mysql buffer_pool_size can improve it but meaningless to adjust to deal with this problem.
-
- Atomicorp Staff - Site Admin
- Posts: 8355
- Joined: Wed Dec 31, 1969 8:00 pm
- Location: earth
- Contact:
Re: IP whitelist increases the server load
2.6.32's netfilter is somewhat dated, in ASL if we do not detect the newer system it will automatically fall back on the older iptables interface which isnt as fast as the newer components. For example a large geo-blacklist (say 200k entries) takes several minutes if not longer on even an idle system, resulting in both high memory overhead and high disk IO. Additionally this method results in a gap period where the firewall policy is esentially open while the policy is loading.
The newer method (using sets and hashes) can not only load the entire geo-db (300k entries) in a fraction of a millisecond, it also lets us swap sets. So you can have a new set loading on an update, and swap places with it without any kind of open state in the firewal policy at all. Its considerably more advanced, and when you get into extremely large firewall policies its really the only way to go.
So could it be the firewall policy? Possibly, and I know for a fact that mysql can be be blocking here as well. It could be a combination, or something else entirely. Until we get a chance to look at it I'm speculating here. We do have one case open to look at a system reporting this, so we'll probably start there. If you don't have a case open already, go ahead and open one up if its something we can look at first hand.
The newer method (using sets and hashes) can not only load the entire geo-db (300k entries) in a fraction of a millisecond, it also lets us swap sets. So you can have a new set loading on an update, and swap places with it without any kind of open state in the firewal policy at all. Its considerably more advanced, and when you get into extremely large firewall policies its really the only way to go.
So could it be the firewall policy? Possibly, and I know for a fact that mysql can be be blocking here as well. It could be a combination, or something else entirely. Until we get a chance to look at it I'm speculating here. We do have one case open to look at a system reporting this, so we'll probably start there. If you don't have a case open already, go ahead and open one up if its something we can look at first hand.
Re: IP whitelist increases the server load
Thank you for explaining Scott. I have a few valid concerns, though:
1. cPanel/CentOS 6.x are all running 2.6.32.x kernels. So if ASL claims it is compatible with cPanel, it must support these kernels efficiently. Even if you add exceptions in your scripts to reduce some of the logic and avoid entering high IO state.
2. ASL was working fine until a few months ago. I confirm we were extensively using asl -bl [IP] and asl -wl [IP] before without any issues.
And lastly, I am unable to access the support portal, as your password reset link resets everything else but the login password. This is the reason I am searching your forums for help.
1. cPanel/CentOS 6.x are all running 2.6.32.x kernels. So if ASL claims it is compatible with cPanel, it must support these kernels efficiently. Even if you add exceptions in your scripts to reduce some of the logic and avoid entering high IO state.
2. ASL was working fine until a few months ago. I confirm we were extensively using asl -bl [IP] and asl -wl [IP] before without any issues.
And lastly, I am unable to access the support portal, as your password reset link resets everything else but the login password. This is the reason I am searching your forums for help.
Re: IP whitelist increases the server load
No they arent, im running the 3.2.62-74 ASL kernel with Centos and Cpanel. Cpanel doesnt include a kernel or care what kernel you use.1. cPanel/CentOS 6.x are all running 2.6.32.x kernels.
For what its worth, im not seeing any issues with the newer kernels.
If everything was easy, then the world wouldn't need engineers.
-
- Atomicorp Staff - Site Admin
- Posts: 8355
- Joined: Wed Dec 31, 1969 8:00 pm
- Location: earth
- Contact:
Re: IP whitelist increases the server load
Believe me if we could make 2.6.18 or 2.6.32's firewall more efficient we would, but there is honestly nothing more that can be done here.
You can email your support requests to support@atomicorp.com too, and bring up that you cant log into the portal there.
You can email your support requests to support@atomicorp.com too, and bring up that you cant log into the portal there.
- mikeshinn
- Atomicorp Staff - Site Admin
- Posts: 4149
- Joined: Thu Feb 07, 2008 7:49 pm
- Location: Chantilly, VA
Re: IP whitelist increases the server load
Is there any reason folks arent using the new kernels? 2.6.32 is ancient, and 2.6.18 is almost in the stone age compared to the improvements in 3.x.
Michael Shinn
Atomicorp - Security For Everyone
Atomicorp - Security For Everyone
-
- Forum User
- Posts: 86
- Joined: Wed Oct 03, 2012 2:51 pm
- Location: Algiers
Re: IP whitelist increases the server load
In our case, we use CloudLinux (Kernel: 2.6.32-531.29.2.lve1.3.11.1.el6.x86_64)mikeshinn wrote:Is there any reason folks arent using the new kernels? 2.6.32 is ancient, and 2.6.18 is almost in the stone age compared to the improvements in 3.x.
It is impossible for us to do without CloudLinux especially for its resource management functionality per account (including mysql-governor).
Re: IP whitelist increases the server load
Same here. Cloudlinux wins in terms of performance for the shared hosting, as some customers abuse resources of the whole server.copernic2006 wrote:In our case, we use CloudLinux (Kernel: 2.6.32-531.29.2.lve1.3.11.1.el6.x86_64)mikeshinn wrote:Is there any reason folks arent using the new kernels? 2.6.32 is ancient, and 2.6.18 is almost in the stone age compared to the improvements in 3.x.
It is impossible for us to do without CloudLinux especially for its resource management functionality per account (including mysql-governor).
Re: IP whitelist increases the server load
It worth mentioning that your 3.x kernels do not work in xenserver environments. VM simply not booting after changing the kernel to 3.2.x (at least for CentOS 6.x)
We have just upgraded to the latest XenServer 6.5 which is based on 3.2.x kernel for Dom0 and has improved support for 3.x kernels in DomU, but still no luck with your kernels.
We have just upgraded to the latest XenServer 6.5 which is based on 3.2.x kernel for Dom0 and has improved support for 3.x kernels in DomU, but still no luck with your kernels.
-
- Atomicorp Staff - Site Admin
- Posts: 8355
- Joined: Wed Dec 31, 1969 8:00 pm
- Location: earth
- Contact:
Re: IP whitelist increases the server load
There is a xen specific kernel channel you need to use there, called tortix-kernel-xen. This is what we use on AWS and Rackspace.
Re: IP whitelist increases the server load
why do they use such an old kernel? looking onkernel.org its no longer supported or maintained.CloudLinux (Kernel: 2.6.32-531.29.2.lve1.3.11.1.el6.x86_64)
If everything was easy, then the world wouldn't need engineers.