Mysql out of memory attacks

isornoserver · Fri Jul 26, 2013 10:21 pm

Hello,

Yesterday we have installed ASL to our CentOS 6 64bit Whm cPanel Server with 4GB ram with 16GB swap ram and after we are facing strange problem it seems someone is attacking our server.

we are receiving server down notifications from pingdom every 2-3 hours and our our server is out of memory and website is taking forever to load.

when we asked to investigate the issue with our hosting provider softlayer he has replied this:

The load average has a 15 minute average of over 11.

{Jul 25 23:35 PM} [192] root@server ~ # uptime
23:35:01 up 20:15, 2 users, load average: 8.54, 10.78, 11.49
{Jul 25 23:35 PM} [193] root@server ~ #

This is due to being almost completely out of memory and dipping extensively in to swap space.

{Jul 25 23:35 PM} [194] root@server ~ # free -m
total used free shared buffers cached
Mem: 3858 3771 86 0 1 39
-/+ buffers/cache: 3730 127
Swap: 16383 14355 2028

I would get with your website's developer about optimizing the mysql queries your site is running as this is most definitely a strong contributing factor.

{Jul 25 23:35 PM} [195] root@server ~ # mysqladmin proc stat
+-------+--------------+-----------------+--------------+---------+------+-------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-------+--------------+-----------------+--------------+---------+------+-------+------------------+
| 79 | eximstats | localhost | eximstats | Sleep | 2148 | | |
| 22963 | tortix | localhost:33234 | tortix | Sleep | 114 | | |
| 28632 | leechprotect | localhost | leechprotect | Sleep | 1294 | | |
| 28752 | tortix | localhost:36675 | tortix | Sleep | 316 | | |
| 28799 | dev1979_u | localhost | dev1979_db | Sleep | 0 | | |
| 28800 | dev1979_u | localhost | dev1979_db | Sleep | 1 | | |
| 28801 | root | localhost | | Query | 0 | | show processlist |
+-------+--------------+-----------------+--------------+---------+------+-------+------------------+
Uptime: 72874 Threads: 7 Questions: 4459582 Slow queries: 60 Opens: 1231 Flush tables: 1 Open tables: 400 Queries per second avg: 61.195

There appear to be lots of blocked processes as well--

{Jul 25 23:37 PM} [197] root@server ~ # vmstat -S M 1 10
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 7 16052 83 1 39 0 0 209 112 28 22 1 0 95 3 0
0 6 16067 83 1 39 1 16 3296 16776 3085 1330 0 2 51 47 0
0 7 16073 84 1 39 0 6 xxxxxxxxxxxxxxxx - CC_FILTER 0 1 62 36 0
0 7 16075 84 1 39 1 2 xxxxxxxxxxxxxxxx - CC_FILTER 0 1 64 35 0
0 12 16078 85 1 39 1 3 2304 3244 754 445 0 1 65 34 0
0 15 16082 84 1 38 1 4 xxxxxxxxxxxxxxxx - CC_FILTER 0 2 44 54 0
0 8 16087 84 1 38 1 5 xxxxxxxxxxxxxxxx - CC_FILTER 0 1 57 42 0
0 8 16094 85 1 36 1 7 xxxxxxxxxxxxxxxx - CC_FILTER 1 1 67 31 0
0 6 16103 84 1 35 1 9 xxxxxxxxxxxxxxxx - CC_FILTER 0 1 53 46 0
1 3 16116 83 1 36 0 13 1280 13888 2709 1074 1 2 76 22 0

The script consuming the most cpu (as well as 0.6% of the memory) appears to be category.php .

{Jul 25 23:39 PM} [200] root@server ~ # ps fuxa | head -1 && ps_fuxa_sorted_by_mem | tail -20
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 30241 0.1 0.0 0 0 ? S 21:40 0:07 \_ [kworker/3:0]
tortix 25141 0.0 0.0 411004 4 ? S 23:06 0:00 \_ /var/asl/usr/sbin/tortixd
tortix 27937 0.0 0.0 410512 8 ? S 23:29 0:00 \_ /var/asl/usr/sbin/tortixd
tortix 28000 0.0 0.0 408860 0 ? S 23:30 0:00 \_ /var/asl/usr/sbin/tortixd
tortix 28056 0.0 0.0 408860 0 ? S 23:31 0:00 \_ /var/asl/usr/sbin/tortixd
ossec 3029 0.2 0.1 17656 5336 ? S 21:53 0:16 /var/ossec/bin/ossec-analysisd
tortix 27997 0.0 0.2 416916 10804 ? S 23:30 0:00 \_ /var/asl/usr/sbin/tortixd
root 2866 0.0 0.5 150736 22204 ? Ss 03:20 0:12 /usr/local/apache/bin/httpd -k start -DSSL
dev1979 28999 13.0 0.6 317184 24196 ? S 23:39 0:00 | \_ /usr/bin/php /home/dev1979/public_html/category.php
nobody 28851 0.0 0.9 151500 37996 ? S 23:38 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28126 0.0 1.0 152060 43116 ? S 23:32 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28795 0.1 1.0 151796 40428 ? S 23:36 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28800 0.0 1.0 151788 40376 ? S 23:37 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28850 0.1 1.0 151940 40652 ? S 23:38 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28857 0.1 1.0 151788 40296 ? S 23:38 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28926 0.1 1.0 151860 43224 ? S 23:39 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28703 0.1 1.1 152976 45044 ? S 23:35 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28774 0.1 1.1 151852 43644 ? S 23:36 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28848 0.1 1.1 151860 45272 ? S 23:38 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
mysql 3662 1.7 2.4 2989488 98656 ? Sl 03:21 21:34 \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/lib/mysql/server.isorno.com.pid
{Jul 25 23:39 PM} [201] root@server ~ #

Once again, you will need to speak to your server administrator/developer about optimizing this.

Thank you for choosing SoftLayer [An IBM company]!

Pingdom report:

PingdomAlert DOWN:
isorno.com (website) is down since 2013-07-27 00:06:28.

PingdomAlert UP:
isorno.com (website) is UP again at 2013-07-27 00:21:28, after 15m of downtime.

PingdomAlert DOWN:
isorno.com (website) is down since 2013-07-27 01:46:28.

PingdomAlert UP:
isorno.com (website) is UP again at 2013-07-27 02:01:28, after 15m of downtime.

PingdomAlert DOWN:
isorno.com (website) is down since 2013-07-27 05:06:28.

PingdomAlert UP:
isorno.com (website) is UP again at 2013-07-27 05:21:28, after 15m of downtime.

PingdomAlert DOWN:
isorno.com (website) is down since 2013-07-27 06:01:28.

PingdomAlert UP:
isorno.com (website) is UP again at 2013-07-27 06:11:28, after 10m of downtime.

PingdomAlert DOWN:
isorno.com (website) is down since 2013-07-27 09:21:28.

PingdomAlert UP:
isorno.com (website) is UP again at 2013-07-27 09:31:28, after 10m of downtime.

As you can see there is some kind of attack is going on our sever. Please let us know what is causing this all.

Best Regards,
Dev

isornoserver · Fri Jul 26, 2013 10:24 pm

21:12:22 server 10 30220 [error] [client 168.63.12.102] request failed: error reading the header
17:33:39 server 7 331030 server.isorno.com
94.23.45.14 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
14:15:59 server 12 5108 server kernel: Out of memory: Kill process 16478 (iptables) score 897 or sacrifice chil
14:10:33 server 12 5108 server kernel: Out of memory: Kill process 16204 (iptables) score 896 or sacrifice chil
14:01:21 server 7 553 File `/etc/rc.d/rc3.d/S61clamd` was deleted. Unable to retrieve checksum
10:43:09 server 7 551 Integrity checksum changed for: `/opt/nimsoft/probes/service/hdb/hdb.logSize changed from `1683` to `1870
10:43:09 server 7 551 Integrity checksum changed for: `/opt/nimsoft/pids/nimbus-0.pids
10:43:08 server 7 551 Integrity checksum changed for: `/opt/nimsoft/robot/controller.logSize changed from `18544` to `20163
10:43:03 server 7 551 Integrity checksum changed for: `/opt/nimsoft/robot/nimbus.logSize changed from `4428` to `4977
10:43:03 server 7 551 Integrity checksum changed for: `/opt/nimsoft/robot/robot.pem
10:43:03 server 7 551 Integrity checksum changed for: `/opt/nimsoft/robot/pids/nimbus.pid
10:43:03 server 7 551 Integrity checksum changed for: `/opt/nimsoft/robot/pids/nimbus-30.pid
10:21:55 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:54 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:54 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:54 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:54 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:54 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:54 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:54 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:54 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:54 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:54 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:53 server 7 331030 server.isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:53 server 7 331030 server.isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:53 server 7 331030 server.isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:53 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:53 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:53 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:53 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:53 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
09:51:39 server 10 30220 [error] [client 168.63.12.102] request failed: error reading the header
08:17:11 server 12 5108 server kernel: Out of memory: Kill process 28746 (iptables) score 885 or sacrifice chil
07:11:27 server 7 331030 isorno.com
50.7.161.106 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
06:17:04 server 7 550 Integrity checksum changed for: `/etc/named.conf.zonedir.cache
06:17:04 server 7 550 Integrity checksum changed for: `/etc/named.conf.cacheSize changed from `213` to `285
06:14:58 server 10 30220 [error] [client 168.63.12.102] request failed: error reading the header
06:09:32 server 7 550 Integrity checksum changed for: `/usr/local/lib/php.ini,vSize changed from `47778` to `49228Permissions changed from `rw-r--r--` to `r--r--r--
06:08:12 server 7 551 Integrity checksum changed for: `/usr/local/lib/php.iniSize changed from `12288` to `37385
05:24:04 server 7 550 Integrity checksum changed for: `/usr/lib/php.ini
05:11:43 server 7 550 Integrity checksum changed for: `/opt/nimsoft/probes/service/hdb/hdb.logSize changed from `1496` to `1683
05:11:43 server 7 550 Integrity checksum changed for: `/opt/nimsoft/pids/nimbus-0.pids
05:11:42 server 7 550 Integrity checksum changed for: `/opt/nimsoft/robot/nimbus.logSize changed from `3879` to `4428
05:11:42 server 7 550 Integrity checksum changed for: `/opt/nimsoft/robot/robot.pem
05:11:42 server 7 550 Integrity checksum changed for: `/opt/nimsoft/robot/pids/nimbus.pid
05:11:42 server 7 550 Integrity checksum changed for: `/opt/nimsoft/robot/pids/nimbus-30.pid
05:11:42 server 7 550 Integrity checksum changed for: `/opt/nimsoft/robot/controller.logSize changed from `16925` to `18544
04:05:31 server 7 552 Integrity checksum changed for: `/etc/asl/rulesSize changed from `118` to `151
04:05:26 server 7 552 Integrity checksum changed for: `/etc/asl/VERSIONSize changed from `213` to `224

log entries from ASL.

isornoserver · Fri Jul 26, 2013 10:46 pm

We are using our server to host our eCommerce website and from past 2 months our website is suffering from hacking attacks and so our orders are being compromised.

we were not able to identify the exact cause of hack but we know hacker is attacking our mysql to compromised our oders that is why we have installed your security solution to avoid hacks and now hacker is overloading our server to shut down.

We have even blocked all countries to avoid attacks but still hacker has found some way to over load mysql so please investigate this issue and let me know if you need further information from our server. Thanks

isornoserver · Unread post by **isornoserver** » Sat Jul 27, 2013 1:47 am

We are receiving emails in every 30 min from pingdom about server down. This is really serious problem and this has happened after we have installed ASL so please provide your support.

Awaiting your soonest response.

isornoserver · Unread post by **isornoserver** » Sat Jul 27, 2013 1:54 am

27 July
00:43:15 server 12 5108 server kernel: Out of memory: Kill process 19958 (iptables) score 899 or sacrifice chil
00:41:53 server 7 553 File `/usr/local/lib/mysql.sock` was deleted. Unable to retrieve checksum
00:32:50 server 7 552 Integrity checksum changed for: `/etc/asl/geo-blacklistSize changed from `705` to `0What changed1,234d< a< a< a< a< a< a< a< a< a< a< a< a< a< a< a< a
00:01:25 server 7 552 Integrity checksum changed for: `/etc/sysconfig/rkhunterSize changed from `338` to `0
26 July
23:10:38 server 7 551 Integrity checksum changed for: `/etc/shadow-What changed1c< root:$1$zdT0Ewn/$53xReQigdC0aml48sdher.

0:99999:7::--> root:$1$OzPbgQuu$3pc.t0Kmg.KaNQF7T5QCL1

0:99999:7::40c4< dev1979:$1$YYMDRTsE$IfNbSbKbG06RQN2Xvx8gD1

0:99999:7::--> dev1979:$1$ASiRKqM3$S.Y8XZAce4Cc0IYYLXET1/

0:99999:7::
22:27:06 server 12 5108 server kernel: Out of memory: Kill process 26923 (iptables) score 882 or sacrifice chil
21:12:22 server 10 30220 [error] [client 168.63.12.102] request failed: error reading the header
17:33:39 server 7 331030 server.isorno.com
94.23.45.14 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
14:15:59 server 12 5108 server kernel: Out of memory: Kill process 16478 (iptables) score 897 or sacrifice chil
14:10:33 server 12 5108 server kernel: Out of memory: Kill process 16204 (iptables) score 896 or sacrifice chil
14:01:21 server 7 553 File `/etc/rc.d/rc3.d/S61clamd` was deleted. Unable to retrieve checksum
10:43:09 server 7 551 Integrity checksum changed for: `/opt/nimsoft/probes/service/hdb/hdb.logSize changed from `1683` to `1870
10:43:09 server 7 551 Integrity checksum changed for: `/opt/nimsoft/pids/nimbus-0.pids
10:43:08 server 7 551 Integrity checksum changed for: `/opt/nimsoft/robot/controller.logSize changed from `18544` to `20163
10:43:03 server 7 551 Integrity checksum changed for: `/opt/nimsoft/robot/nimbus.logSize changed from `4428` to `4977
10:43:03 server 7 551 Integrity checksum changed for: `/opt/nimsoft/robot/robot.pem
10:43:03 server 7 551 Integrity checksum changed for: `/opt/nimsoft/robot/pids/nimbus.pid
10:43:03 server 7 551 Integrity checksum changed for: `/opt/nimsoft/robot/pids/nimbus-30.pid
10:21:55 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:54 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:54 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:54 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:54 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:54 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:54 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:54 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:54 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:54 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:54 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:53 server 7 331030 server.isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:53 server 7 331030 server.isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:53 server 7 331030 server.isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:53 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:53 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:53 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:53 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header
10:21:53 server 7 331030 isorno.com
213.229.74.196 Atomicorp.com WAF Rules: Suspicious activity detected - HTTP Request Missing a Host Header

isornoserver · Unread post by **isornoserver** » Sat Jul 27, 2013 3:17 am

I have found some mysql my.cnf file settings to improve mysql performance:

[mysqld]
back_log = 75
innodb_buffer_pool_size = 512M
innodb_additional_mem_pool_size = 10M
table_open_cache = 1024
join_buffer_size = 40M
query_cache_size = 128M
query_cache_limit = 4M
table_definition_cache=256
max_connections = 300
log_slow_queries = /var/log/mysql/mysql-slow.log
long_query_time = 3
key_buffer = 384M
myisam_sort_buffer_size = 64M
read_buffer_size = 1M
sort_buffer_size = 2M
table_cache = 1800
thread_cache_size = 384
wait_timeout = 7200
connect_timeout = 10
tmp_table_size = 64M
max_heap_table_size = 64M
max_allowed_packet = 64M
max_connect_errors = 1000
read_rnd_buffer_size = 524288
bulk_insert_buffer_size = 8M
query_cache_type = 1
query_prealloc_size = 65536
query_alloc_block_size = 131072
default-storage-engine = InnoDB

[mysqld_safe]
nice = -5
open_files_limit = 8192

[mysqldump]
quick
max_allowed_packet = 16M

[myisamchk]
key_buffer = 64M
sort_buffer = 64M
read_buffer = 16M
write_buffer = 16M

Please let me know if above settings can solve the issue. Thanks

isornoserver · Unread post by **isornoserver** » Sat Jul 27, 2013 3:32 am

[mysqld]
back_log = 75
innodb_buffer_pool_size = 512M
innodb_additional_mem_pool_size = 10M
table_open_cache = 1024
join_buffer_size = 40M
query_cache_size = 128M
query_cache_limit = 4M
table_definition_cache=256
max_connections = 300
log_slow_queries = /var/log/mysql/mysql-slow.log
long_query_time = 3
key_buffer = 384M
myisam_sort_buffer_size = 64M
read_buffer_size = 1M
sort_buffer_size = 2M
table_cache = 1800
thread_cache_size = 384
wait_timeout = 7200
connect_timeout = 10
tmp_table_size = 64M
max_heap_table_size = 64M
max_allowed_packet = 64M
max_connect_errors = 1000
read_rnd_buffer_size = 524288
bulk_insert_buffer_size = 8M
query_cache_type = 1
query_prealloc_size = 65536
query_alloc_block_size = 131072
default-storage-engine = InnoDB
log-warnings=2
symbolic-links=0
innodb_file_per_table=1
local-infile=0

[mysqld_safe]
nice = -5
open_files_limit = 8192
log-error=/var/log/mysqld.log

[mysqldump]
quick
max_allowed_packet = 16M

[myisamchk]
key_buffer = 64M
sort_buffer = 64M
read_buffer = 16M
write_buffer = 16M

isornoserver · Unread post by **isornoserver** » Sat Jul 27, 2013 4:20 am

Our hosting company provide more information:

iptables is using 76% or your memory. Please review your iptables rules.

top - 03:10:41 up 17:40, 3 users, load average: 12.92, 10.37, 5.97
Tasks: 195 total, 1 running, 191 sleeping, 3 stopped, 0 zombie
Cpu(s): 0.2%us, 1.3%sy, 0.0%ni, 21.0%id, 77.2%wa, 0.0%hi, 0.3%si, 0.0%st
Mem: 3950936k total, 3864176k used, 86760k free, 1788k buffers
Swap: 16777212k total, 16106228k used, 670984k free, 42436k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
29364 root 20 0 17.4g 2.9g 312 D 1.7 76.3 0:40.07 iptables
22699 mysql 15 -5 2197m 54m 3528 S 0.0 1.4 1:12.09 mysqld
30744 root 20 0 67840 48m 696 D 1.0 1.3 0:00.13 iptables
29822 tortix 20 0 401m 44m 4252 S 0.0 1.2 0:00.74 tortixd
29450 tortix 20 0 401m 44m 4312 S 0.0 1.2 0:01.95 tortixd

Thank you for choosing SoftLayer.

Please let us know know how to fix the issue.

BruceLee · Unread post by **BruceLee** » Sat Jul 27, 2013 3:32 pm

I don't think that I can help you a lot but I do my best and have some questions, comments and info from my side.
1. If you urgently need help use the support portal not the forum
2. If you need to get it running asap disable ASL and reboot to non ALS kernel
3. iptables is now consuming most memory. before that it was mysql. currently I don't see any reason that ASL is the culprit. But maybe some misconfiguration.
4. What binaries do you have installed to manage iptables. apf or something else maybe?
5. post the iptables ruleset

isornoserver · Sun Jul 28, 2013 10:38 pm

Hi BruceLee,

We have reinstalled the whole system in this order to avoid any misconfiguration :

1. Install CentOs
2. Install Cpanel
3. Rebuild apache without mod security
4. Install ASL.

we haven't changed ASL default settings also but still our server getting down every 1-2 hours for 10-15min due to iptables are consuming over 75% of ram.

as its a fresh installed server so iptables ruleset should contain default configuration as we have not modified any thing.

This is log report from ASL and it clearly shows server is Out of memory due to iptables.

28 July
21:23:02 server 7 551 Integrity checksum changed for: `/etc/asl/VERSIONSize changed from `224` to `213
21:06:48 server 7 550 Integrity checksum changed for: `/etc/asl/VERSIONWhat changed6c< MODSEC_VERSION=20130726144--> MODSEC_VERSION=20130728125
21:06:23 server 7 2932 server yum[7235]: Installed: xtables-addons-1.47.1-3.36.el6.x86_6
21:06:18 server 7 2932 server yum[7235]: Installed: kmod-xtables-addons-3.2.48-54.art.x86_64-1.47.1-3.x86_6
21:06:18 server 7 550 Integrity checksum changed for: `/etc/ld.so.cacheSize changed from `52053` to `52156
21:06:18 server 7 553 File `/etc/rc.d/rc0.d/K88iscsi` was deleted. Unable to retrieve checksum
21:06:17 server 7 550 Integrity checksum changed for: `/lib/modules/3.2.48-54.art.x86_64/modules.depSize changed from `221604` to `223881
21:06:17 server 7 550 Integrity checksum changed for: `/lib/modules/3.2.48-54.art.x86_64/modules.dep.binSize changed from `321417` to `324941
21:06:17 server 7 550 Integrity checksum changed for: `/lib/modules/3.2.48-54.art.x86_64/modules.aliasSize changed from `612567` to `613735
21:06:17 server 7 550 Integrity checksum changed for: `/lib/modules/3.2.48-54.art.x86_64/modules.alias.binSize changed from `599542` to `602012
21:06:17 server 7 550 Integrity checksum changed for: `/lib/modules/3.2.48-54.art.x86_64/modules.symbolsSize changed from `224936` to `225428
21:06:17 server 7 550 Integrity checksum changed for: `/lib/modules/3.2.48-54.art.x86_64/modules.symbols.binSize changed from `283363` to `284115
21:05:26 server 7 551 Integrity checksum changed for: `/etc/asl/configSize changed from `9420` to `0What changed199c19< # PSMON configuration--> # PSMON configuration.
21:05:26 server 7 552 Integrity checksum changed for: `/etc/asl/configSize changed from `0` to `9421
21:01:04 server 7 552 Integrity checksum changed for: `/etc/yum.conf
19:01:20 server 7 553 File `/etc/rc.d/rc3.d/S61clamd` was deleted. Unable to retrieve checksum
16:01:01 server 7 551 Integrity checksum changed for: `/etc/yum.conf
15:01:04 server 7 550 Integrity checksum changed for: `/etc/yum.confSize changed from `1184` to `0
14:02:54 server 12 5108 server kernel: Out of memory: Kill process 18226 (iptables) score 879 or sacrifice chil
12:30:46 server 7 552 Integrity checksum changed for: `/opt/nimsoft/robot/spooler.logSize changed from `1810` to `1888
12:10:43 server 7 550 Integrity checksum changed for: `/opt/nimsoft/robot/q1.rdbSize changed from `0` to `571
12:10:43 server 7 551 Integrity checksum changed for: `/opt/nimsoft/robot/q1.rdbSize changed from `571` to `1128
12:10:43 server 7 552 Integrity checksum changed for: `/opt/nimsoft/robot/q1.rdbSize changed from `1128` to `1687
12:05:42 server 7 550 Integrity checksum changed for: `/opt/nimsoft/robot/q2.rdbSize changed from `0` to `571
12:05:42 server 7 551 Integrity checksum changed for: `/opt/nimsoft/robot/q2.rdbSize changed from `571` to `1128
12:05:42 server 7 552 Integrity checksum changed for: `/opt/nimsoft/robot/q2.rdbSize changed from `1128` to `1687
12:01:06 server 7 550 Integrity checksum changed for: `/etc/fstabSize changed from `1250` to `1255What changed16c1< tmpfs /dev/shm tmpfs defaults 0 --> tmpfs /dev/shm tmpfs noexec,nosuid 0
12:01:06 server 7 551 Integrity checksum changed for: `/opt/nimsoft/robot/spooler.logSize changed from `1732` to `1810
11:54:19 server 7 510 File `/etc/mime.types` is owned by root and has written permissions to anyone
11:54:19 server 7 510 File `/usr/local/cpanel/3rdparty/php/53/lib64/pear/HTTP_Request.xml` is owned by root and has written permissions to anyone
11:54:19 server 7 510 File `/usr/local/cpanel/3rdparty/php/53/lib64/pear/Date.xml` is owned by root and has written permissions to anyone
11:54:19 server 7 510 File `/usr/local/cpanel/3rdparty/php/53/lib64/pear/HTTP.xml` is owned by root and has written permissions to anyone
11:54:19 server 7 510 File `/usr/local/cpanel/3rdparty/php/53/lib64/pear/HTML_Template_IT.xml` is owned by root and has written permissions to anyone
11:07:07 server 7 550 Integrity checksum changed for: `/usr/bin/x86_64-redhat-linux-gccPermissions changed from `rwxr-xr-x` to `rwxr-x---Group ownership was `0`, now it is `32007
11:06:27 server 7 550 Integrity checksum changed for: `/usr/bin/g++Permissions changed from `rwxr-xr-x` to `rwxr-x---Group ownership was `0`, now it is `32007
11:05:16 server 7 550 Integrity checksum changed for: `/usr/bin/c99Permissions changed from `rwxr-xr-x` to `rwxr-x---Group ownership was `0`, now it is `32007
11:04:36 server 7 550 Integrity checksum changed for: `/usr/bin/x86_64-redhat-linux-c++Permissions changed from `rwxr-xr-x` to `rwxr-x---Group ownership was `0`, now it is `32007
11:04:21 server 7 550 Integrity checksum changed for: `/usr/bin/c89Permissions changed from `rwxr-xr-x` to `rwxr-x---Group ownership was `0`, now it is `32007
11:04:16 server 7 550 Integrity checksum changed for: `/usr/bin/gccPermissions changed from `rwxr-xr-x` to `rwxr-x---Group ownership was `0`, now it is `32007
11:04:06 server 7 550 Integrity checksum changed for: `/usr/bin/ldPermissions changed from `rwxr-xr-x` to `rwxr-x---Group ownership was `0`, now it is `32007
11:03:35 server 7 550 Integrity checksum changed for: `/usr/bin/c++Permissions changed from `rwxr-xr-x` to `rwxr-x---Group ownership was `0`, now it is `32007
11:00:40 server 7 550 Integrity checksum changed for: `/usr/bin/x86_64-redhat-linux-g++Permissions changed from `rwxr-xr-x` to `rwxr-x---Group ownership was `0`, now it is `32007
10:56:39 server 7 550 Integrity checksum changed for: `/opt/nimsoft/probes/system/processes/processes.logSize changed from `520` to `780
10:56:39 server 7 550 Integrity checksum changed for: `/opt/nimsoft/probes/service/hdb/hdb.logSize changed from `309` to `496
10:56:39 server 7 550 Integrity checksum changed for: `/opt/nimsoft/pids/nimbus-0.pids
10:56:34 server 7 550 Integrity checksum changed for: `/opt/nimsoft/probes/system/cdm/cdm.logSize changed from `532` to `790
10:56:29 server 7 550 Integrity checksum changed for: `/opt/nimsoft/robot/controller.cfgSize changed from `1842` to `1674
10:56:24 server 7 550 Integrity checksum changed for: `/opt/nimsoft/robot/pids/nimbus-30.pid
10:56:24 server 7 550 Integrity checksum changed for: `/opt/nimsoft/robot/pids/nimbus.pid
10:56:24 server 7 550 Integrity checksum changed for: `/opt/nimsoft/robot/nimbus.logSize changed from `791` to `1340

I'm newbie so please provide commands if you need more information from our server. thanks

BruceLee · Unread post by **BruceLee** » Mon Jul 29, 2013 3:55 am

examine your iptables rules. Maybe you can pinpoint it down to something.
manpage is here:
http://ipset.netfilter.org/iptables.man.html

Also there is a chance that you server is being synflooded.
Hard to say.
I would definitely try to go step by step.
Since you have a clean install check under which conditions cpanel on/off, asl featues on/off etc. the memory usage increases.
What do the logs say.
Is this a VPS? Maybe 4GB is not enough. Do you have guaranteed 4GB?
Also you say that you have 4Gb RAM and 16GB Swap RAM?
Sounds like a weird config. I would go with the 20GB Ram and put swap on disk.

hostingg · Unread post by **hostingg** » Mon Jul 29, 2013 1:15 pm

You're hosting company says iptables is using up most of your memory? I dont think they know what they are talking about, iptables doesnt use memory, its just a tool, it runs, it quits. It doesnt use memory.

This is a bit like a mechanic telling you that you need a new car because all your tires are out of gasoline. Thats the sign of a really bad mechanic thats making things up.

So you may want to ask your hosting company to get a more experienced engineer involved to explain whats really happening on your system. I can assure you, its not iptables. More than likely you have a script or web application thats running your server out of memory.

Unread post by **scott** » Mon Jul 29, 2013 6:24 pm

Yeah I cant think of any way the real iptables binary would do this. Its really just an interface to the firewall which is called netfilter. Is it possible yours is modified?

isornoserver · Unread post by **isornoserver** » Tue Jul 30, 2013 5:59 am

Hello all,

I'm newbie and not experienced in linux, please provide me detailed information about technical points so that i can understand it and also provide you with enough information from my server if required.

I have purchased dedicated server with hardware firewall from softlayer with below configuration:

Motherboard SuperMicro X9SCI-LN4F Intel Xeon SingleProc SATA [1Proc]
Processor Intel Xeon-SandyBridge E3-1270-Quadcore [3.4GHz] Hardware upgrade
RAM slot 1 Samsung 2GB DDR3 x8 [2GB] Hardware upgrade
RAM slot 2 Samsung 2GB DDR3 x8 [2GB] Hardware upgrade
RAM slot 3 Empty Hardware upgrade
RAM slot 4 Empty Hardware upgrade

Drive Controller Mainboard \ Onboard \ SATAII Controller Hardware upgrade
Hard Drive 1 wmayp2032323 Western Digital WD Caviar RE4 WD5003ABYX [500GB] Hardware upgrade
Hard Drive 2 Empty Hardware upgrade
Hard Drive 3 Empty Hardware upgrade
Hard Drive 4 Empty Hardware upgrade

Remote Mgmt Card SuperMicro Nuvoton WPCM450 - Onboard IPMI-KVM
Power Supply SuperMicro PWS-605P-1H 600W
Backplane SuperMicro BPN-SAS-815TQ 4 Port Passive

Total server RAM is 4GB and HDD swap drive space is 16GB (swap hdd ram can be changed with reload OS, please let me know if more swap hdd space is required)

You have mentioned that our server is being syn flooded so could you please provide me step by step commands which i can run on my server to gain enough information for you to check if this is the case and also let me know how can i check cpanel and asl on/off services?

i have asked my hosting company to investigate the exact cause for server down as ip tables are not the culprit so once i have their reply i will post here. Thank you.

isornoserver · Unread post by **isornoserver** » Tue Jul 30, 2013 6:33 am

Our hosting provider relied this:

Actually IPtables is a program on the server and does use memory, though it is normally very slim however if your server was under a high attack it would cause iptables to work harder and use more resources. Also as you can cleary see from top iptables was using 76% of memory. This is output directly from the server and is not made up.

PHP is also what was using up the most CPU. At this time we still hold the same recommendation and that is to upgrade your RAM.

Let us know if we can be of further assistance, and thank you for choosing SoftLayer.

Please let me know how to check if our server is under what kind of high attack?

Atomicorp

Mysql out of memory attacks

Mysql out of memory attacks

Re: Mysql out of memory attacks

Re: Mysql out of memory attacks

Re: Mysql out of memory attacks

Re: Mysql out of memory attacks

Re: Mysql out of memory attacks

Re: Mysql out of memory attacks

Re: Mysql out of memory attacks

Re: Mysql out of memory attacks

Re: Mysql out of memory attacks

Re: Mysql out of memory attacks

Re: Mysql out of memory attacks

Re: Mysql out of memory attacks

Re: Mysql out of memory attacks

Re: Mysql out of memory attacks