IP whitelist increases the server load

[copernic2006]

Hello,
I noticed that every time I execute the command asl -wl IPADRESS the serverload value increases rapidly (usually between 1 and 2) and running the command, the value can reach 22.

If someone has an idea of the cause of the problem, I'm interested.

top - 22:43:42 up 31 days, 22:16, 1 user, load average: 20.98, 14.27, 7.30
Tasks: 618 total, 2 running, 611 sleeping, 0 stopped, 5 zombie
Cpu(s): 15.5%us, 13.4%sy, 0.0%ni, 3.1%id, 67.9%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 32879940k total, 8580396k used, 24299544k free, 290416k buffers
Swap: 0k total, 0k used, 0k free, 633116k cached


OS: CLOUDLINUX 6.6 x86_64 standard – alger
Panel: WHM 11.46.0 (build 22)
Apache 2.2
Mysql: 5.6.21

[prupert]

There is a high cpu% of iowait, this means that your cpu is waiting for operations on the disk to be completed. This usually indicates a slow disk and/or very high disk io activity.

The load is an average number of running and queued processes. If this number is higher than your number of CPU cores, this means that on average processes are waiting. The number can sometimes be skewed because of a high number of stacked/locked up processes. 618 processes is quite a lot, so check that as well.

[copernic2006]

Prupert thank you for the explanations.
I'll start by checking the status of the disk since I noticed a slowdown sites.

[scott]

Check out sysdig (available from the atomic repo), it is an outstanding tool for ID'ing the source for this kind of thing

[iv@rh]

Great find. Having exactly the same issue, here is the summary:

- virtualised cPanel (latest) with Cloudlinux and around 50 accounts (not busy server, usually running around 0.30 for 15 minutes load average), never hits swap.
- Usual CPU wait time is below 0.5%, CPU Steal is below 0.1%
- Xenserver 6.2SP1 (latest, fully patched)
- ASL 4
- 8 CPU/16GB RAM
- EXT4 filesystem
- 10GBPS fiber storage backend with NFS storage/ZFS NAS, separate /boot and / partitions.
- No major customisations made to ASL configuration (maybe some WAF rules are disabled and others enabled)
- Other non-cPanel VMs (with and without ASL kernel) are running just fine.

Every time (no exceptions), when blacklisting an IP via SSH (asl -bl IP), server load goes up, no sites are loadable. Server is unusable, the only way to quickly restore it is to force-reboot. Without force-reboot it could be running like that for 2 hours (we had to force-reboot it after 2 hours to restore access).

Here are observations of what is happening:

- Huge read traffic. Xencenter reports around 150 MBPS (Mega Bytes per second).
- NAS console reports storage network (only read) traffic around 1.2GBPS. Disk read throughput and IOPS are as usual, but ZFS is delivering lots of files from ARC (in-memory cache), this may explain why disk throughput does not show huge increase in read traffic.
- NewRelic reports IOPS reaching 10K and going flat there. I feel IOPS maxing out, otherwise NAS is capable of reaching 8+GBPS in data transfer from/to a single VM (this was confirmed, benchmarked using Citrix Performance VM running CentOS 6)
- last time this happened we were even unable to launch iotop, the server was that busy.
- top reveals about 80% wait time, no processes are sitting at the top to give an idea of what might be wrong. The server just slows down without unusual activity in 'top'.
- Inspecting messages log after the hard-reboot does not reveal anything useful.

Please advise what can be done to resolve this. Maybe ASL team can reproduce this in a cPanel server running cloud linux (and cloud linux kernel)? This is a really huge problem for us, also as I see now, there are more ASL users experiencing same issue.

[DarkF@der]

Hello Scott i can't find sysdig i also notice slowdowns i also notice when i switch to the normal kernel the server will be faster then i use the ASL kernel.

[root@serverxx ~]# yum install sysdig
Loaded plugins: fastestmirror
Setting up Install Process
Loading mirror speeds from cached hostfile
* asl-4.0: www6.atomicorp.com
* atomic: mirror1.34sp.com
* base: mirror.prolocation.net
* extras: centos.mirror.triple-it.nl
* tortix: www6.atomicorp.com
* tortix-kernel: www6.atomicorp.com
* updates: mirror.prolocation.net
No package sysdig available.
Error: Nothing to do
[root@serverxx ~]#

[scott]

Its available for EL6 and up, Im guessing you're on EL5?

[DarkF@der]

No i'm using centos 6.5

[iv@rh]

Its available for EL6 and up, Im guessing you're on EL5?

Thank you for reviewing my post Scott.

You are missing the point, when advising using tool to debug this. The point is, that blacklisting/whitelisting an IP in firewall should be a lightweight operation, not involving anything else. I suspect ASL is running some scan after an IP is blacklisted/whitelisted. As an atomicorp staff, are you able to confirm it is (or it is not) the case?

For example, doing the same via the ASL console involves 'asl -s -f' scan. Why? Seem to be an overkill to run 'asl -s -f' scan after every change made to the ASL.

It is just

Code: Select all

iptables -I INPUT -s 1.2.3.4 -j DROP

which is a lightweight operation.

[copernic2006]

iv@rh, to Whitelist IP, asl performs a scan, if you made asl -wl IPadress, you'll notice immediately asl performs a scan (similar to the launch of the asl -s -f)
The last time I execute asl -wl IPADRESS the serveload has climbed to 50 and can not connect to the server (SSH, WHM), I had to do a hardware reboot to be able to connect back to my server .
For the time I disabled the automatic update asl and I avoid using whitelist / blacklist IP.
The last time I used -wl asl, I noticed an error:
GC Warning: Repeated allocation of very large block (appr size 696,320.):
May lead to memory leak and poor performance.

Strangely on a second server, also with cPanel (cloudlinux) and ASL, I encounter any problems.

[scott]

Whitelisting is implemented in 3 places:

1) the firewall (aka: Netfilter)
2) The hids (active response)
3) the WAF (rule level whitelisting).


Each one of these components is going to interact with the system in very different ways when each component is restarted, which is why we need more data. If you're both on cloudlinux, then the first place I would look would be on the firewall. That platform doesnt support the fast firewall system we're using in the ASL kernel, so thats the first place to look.

And sysdig 0.1.89 is going back into the repo now. Turns out the newer sysdig isnt compatible with el6 any more.

[hostingg]

iptables -I INPUT -s 1.2.3.4 -j DROP

which is a lightweight operation.

If you run asl -wl youll see its adding the whitelist entry to the waf (and restarting apache), ossec (and reloading its config so its whitelist is updated) and also adding a rule to the firewall. so its doing more than just adding a simple firewall rule.

[octet]

i also notice slowdowns when i switch to the normal kernel the server will be faster then i use the ASL kernel.

Every time (no exceptions), when blacklisting an IP via SSH (asl -bl IP), server load goes up, no sites are loadable. Server is unusable, the only way to quickly restore it is to force-reboot. Without force-reboot it could be running like that for 2 hours (we had to force-reboot it after 2 hours to restore access). Inspecting messages log after the hard-reboot does not reveal anything useful.

Having the same problems on a daily basis, I need to kill iptables, flush it, kill php-cgi and httpd and restart the apache, all from console as I'm not able to connect via SSH anymore.

It's really frustrating!

[scott]

Ah so its iptables on your system huh. Thats completely different from what we're seeing on one of the other systems in this threat, there its mysql related.

Are you using the ASL kernel? This would let you use the faster firewall system (it does not use iptables at all), and do you have large geo or local blacklists?

[octet]

Hi Scott,

MySQL is also running 200% when the load happens

[root@alien ~]# uname -a
Linux alien.3dev.biz 3.2.60-71.art.x86_64 #1 SMP Mon Jul 7 11:24:28 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux

[root@alien ~]# cat /etc/asl/geo-blacklist
cn

[root@alien ~]# cat /etc/asl/blacklist
68.207.225.131,2014-09-19 12:34:44,,
91.200.12.21,2014-10-02 12:22:38,,
114.215.185.20,2014-09-18 13:46:19,,
114.215.185.20,2014-09-18 13:46:23,,
5.254.98.90,2014-10-13 20:11:17,,


Chain INPUT (policy ACCEPT)
target prot opt source destination
ASL-ACTIVE-RESPONSE all -- 198.52.232.248 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 192.240.220.121 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 110.86.179.68 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 195.211.154.159 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 192.171.250.87 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 69.12.75.216 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 204.44.112.40 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 218.59.238.92 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 93.93.56.4 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 125.7.10.63 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 104.168.53.47 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 104.168.53.43 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 104.168.32.46 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 146.0.78.8 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 85.31.196.141 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 192.187.99.2 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 195.154.211.26 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 76.164.192.43 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 173.44.37.242 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 177.11.51.67 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 109.199.82.5 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 5.39.219.27 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 195.162.69.58 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 146.0.73.133 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 208.51.62.154 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 46.22.217.109 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 193.201.224.18 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 193.201.224.54 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 23.254.131.160 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 37.57.200.106 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 93.118.75.143 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 141.105.66.179 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 37.57.231.125 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 5.231.75.186 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 134.249.51.137 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 159.224.160.6 0.0.0.0/0

Chain FORWARD (policy ACCEPT)
target prot opt source destination
ASL-ACTIVE-RESPONSE all -- 198.52.232.248 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 192.240.220.121 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 110.86.179.68 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 195.211.154.159 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 192.171.250.87 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 69.12.75.216 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 204.44.112.40 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 218.59.238.92 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 93.93.56.4 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 125.7.10.63 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 104.168.53.47 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 104.168.53.43 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 104.168.32.46 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 146.0.78.8 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 85.31.196.141 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 192.187.99.2 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 195.154.211.26 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 76.164.192.43 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 173.44.37.242 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 177.11.51.67 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 109.199.82.5 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 5.39.219.27 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 195.162.69.58 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 146.0.73.133 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 208.51.62.154 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 46.22.217.109 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 193.201.224.18 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 193.201.224.54 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 23.254.131.160 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 37.57.200.106 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 93.118.75.143 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 141.105.66.179 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 37.57.231.125 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 5.231.75.186 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 134.249.51.137 0.0.0.0/0
ASL-ACTIVE-RESPONSE all -- 159.224.160.6 0.0.0.0/0

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain ASL-ACTIVE-RESPONSE (72 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0

Chain ASL-AUTOSHUN (0 references)
target prot opt source destination

Chain ASL-AUTOSHUN-DROP-LOG (0 references)
target prot opt source destination

Chain ASL-CIARMY (0 references)
target prot opt source destination

Chain ASL-CIARMY-DROP-LOG (0 references)
target prot opt source destination

Chain ASL-DSHIELD (0 references)
target prot opt source destination

Chain ASL-DSHIELD-DROP-LOG (0 references)
target prot opt source destination

Chain ASL-ELASSO (0 references)
target prot opt source destination

Chain ASL-ELASSO-DROP-LOG (0 references)
target prot opt source destination

Chain ASL-EMERGING (0 references)
target prot opt source destination

Chain ASL-EMERGING-DROP-LOG (0 references)
target prot opt source destination

Chain ASL-GEO-BLACKLIST (0 references)
target prot opt source destination

Chain ASL-GEO-BLACKLIST-LOG (0 references)
target prot opt source destination

Chain ASL-LASSO (0 references)
target prot opt source destination

Chain ASL-LASSO-DROP-LOG (0 references)
target prot opt source destination

Chain ASL-OPENPROXIES (0 references)
target prot opt source destination

Chain ASL-OPENPROXIES-DROP-LOG (0 references)
target prot opt source destination

Chain ASL-TOR (0 references)
target prot opt source destination

Chain ASL-TOR-DROP-LOG (0 references)
target prot opt source destination