Mysql out of memory attacks
Posted: Fri Jul 26, 2013 10:21 pm
Hello,
Yesterday we have installed ASL to our CentOS 6 64bit Whm cPanel Server with 4GB ram with 16GB swap ram and after we are facing strange problem it seems someone is attacking our server.
we are receiving server down notifications from pingdom every 2-3 hours and our our server is out of memory and website is taking forever to load.
when we asked to investigate the issue with our hosting provider softlayer he has replied this:
The load average has a 15 minute average of over 11.
{Jul 25 23:35 PM} [192] root@server ~ # uptime
23:35:01 up 20:15, 2 users, load average: 8.54, 10.78, 11.49
{Jul 25 23:35 PM} [193] root@server ~ #
This is due to being almost completely out of memory and dipping extensively in to swap space.
{Jul 25 23:35 PM} [194] root@server ~ # free -m
total used free shared buffers cached
Mem: 3858 3771 86 0 1 39
-/+ buffers/cache: 3730 127
Swap: 16383 14355 2028
I would get with your website's developer about optimizing the mysql queries your site is running as this is most definitely a strong contributing factor.
{Jul 25 23:35 PM} [195] root@server ~ # mysqladmin proc stat
+-------+--------------+-----------------+--------------+---------+------+-------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-------+--------------+-----------------+--------------+---------+------+-------+------------------+
| 79 | eximstats | localhost | eximstats | Sleep | 2148 | | |
| 22963 | tortix | localhost:33234 | tortix | Sleep | 114 | | |
| 28632 | leechprotect | localhost | leechprotect | Sleep | 1294 | | |
| 28752 | tortix | localhost:36675 | tortix | Sleep | 316 | | |
| 28799 | dev1979_u | localhost | dev1979_db | Sleep | 0 | | |
| 28800 | dev1979_u | localhost | dev1979_db | Sleep | 1 | | |
| 28801 | root | localhost | | Query | 0 | | show processlist |
+-------+--------------+-----------------+--------------+---------+------+-------+------------------+
Uptime: 72874 Threads: 7 Questions: 4459582 Slow queries: 60 Opens: 1231 Flush tables: 1 Open tables: 400 Queries per second avg: 61.195
There appear to be lots of blocked processes as well--
{Jul 25 23:37 PM} [197] root@server ~ # vmstat -S M 1 10
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 7 16052 83 1 39 0 0 209 112 28 22 1 0 95 3 0
0 6 16067 83 1 39 1 16 3296 16776 3085 1330 0 2 51 47 0
0 7 16073 84 1 39 0 6 xxxxxxxxxxxxxxxx - CC_FILTER 0 1 62 36 0
0 7 16075 84 1 39 1 2 xxxxxxxxxxxxxxxx - CC_FILTER 0 1 64 35 0
0 12 16078 85 1 39 1 3 2304 3244 754 445 0 1 65 34 0
0 15 16082 84 1 38 1 4 xxxxxxxxxxxxxxxx - CC_FILTER 0 2 44 54 0
0 8 16087 84 1 38 1 5 xxxxxxxxxxxxxxxx - CC_FILTER 0 1 57 42 0
0 8 16094 85 1 36 1 7 xxxxxxxxxxxxxxxx - CC_FILTER 1 1 67 31 0
0 6 16103 84 1 35 1 9 xxxxxxxxxxxxxxxx - CC_FILTER 0 1 53 46 0
1 3 16116 83 1 36 0 13 1280 13888 2709 1074 1 2 76 22 0
The script consuming the most cpu (as well as 0.6% of the memory) appears to be category.php .
{Jul 25 23:39 PM} [200] root@server ~ # ps fuxa | head -1 && ps_fuxa_sorted_by_mem | tail -20
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 30241 0.1 0.0 0 0 ? S 21:40 0:07 \_ [kworker/3:0]
tortix 25141 0.0 0.0 411004 4 ? S 23:06 0:00 \_ /var/asl/usr/sbin/tortixd
tortix 27937 0.0 0.0 410512 8 ? S 23:29 0:00 \_ /var/asl/usr/sbin/tortixd
tortix 28000 0.0 0.0 408860 0 ? S 23:30 0:00 \_ /var/asl/usr/sbin/tortixd
tortix 28056 0.0 0.0 408860 0 ? S 23:31 0:00 \_ /var/asl/usr/sbin/tortixd
ossec 3029 0.2 0.1 17656 5336 ? S 21:53 0:16 /var/ossec/bin/ossec-analysisd
tortix 27997 0.0 0.2 416916 10804 ? S 23:30 0:00 \_ /var/asl/usr/sbin/tortixd
root 2866 0.0 0.5 150736 22204 ? Ss 03:20 0:12 /usr/local/apache/bin/httpd -k start -DSSL
dev1979 28999 13.0 0.6 317184 24196 ? S 23:39 0:00 | \_ /usr/bin/php /home/dev1979/public_html/category.php
nobody 28851 0.0 0.9 151500 37996 ? S 23:38 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28126 0.0 1.0 152060 43116 ? S 23:32 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28795 0.1 1.0 151796 40428 ? S 23:36 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28800 0.0 1.0 151788 40376 ? S 23:37 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28850 0.1 1.0 151940 40652 ? S 23:38 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28857 0.1 1.0 151788 40296 ? S 23:38 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28926 0.1 1.0 151860 43224 ? S 23:39 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28703 0.1 1.1 152976 45044 ? S 23:35 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28774 0.1 1.1 151852 43644 ? S 23:36 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28848 0.1 1.1 151860 45272 ? S 23:38 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
mysql 3662 1.7 2.4 2989488 98656 ? Sl 03:21 21:34 \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/lib/mysql/server.isorno.com.pid
{Jul 25 23:39 PM} [201] root@server ~ #
Once again, you will need to speak to your server administrator/developer about optimizing this.
Thank you for choosing SoftLayer [An IBM company]!
Pingdom report:
PingdomAlert DOWN:
isorno.com (website) is down since 2013-07-27 00:06:28.
PingdomAlert UP:
isorno.com (website) is UP again at 2013-07-27 00:21:28, after 15m of downtime.
PingdomAlert DOWN:
isorno.com (website) is down since 2013-07-27 01:46:28.
PingdomAlert UP:
isorno.com (website) is UP again at 2013-07-27 02:01:28, after 15m of downtime.
PingdomAlert DOWN:
isorno.com (website) is down since 2013-07-27 05:06:28.
PingdomAlert UP:
isorno.com (website) is UP again at 2013-07-27 05:21:28, after 15m of downtime.
PingdomAlert DOWN:
isorno.com (website) is down since 2013-07-27 06:01:28.
PingdomAlert UP:
isorno.com (website) is UP again at 2013-07-27 06:11:28, after 10m of downtime.
PingdomAlert DOWN:
isorno.com (website) is down since 2013-07-27 09:21:28.
PingdomAlert UP:
isorno.com (website) is UP again at 2013-07-27 09:31:28, after 10m of downtime.
As you can see there is some kind of attack is going on our sever. Please let us know what is causing this all.
Best Regards,
Dev
Yesterday we have installed ASL to our CentOS 6 64bit Whm cPanel Server with 4GB ram with 16GB swap ram and after we are facing strange problem it seems someone is attacking our server.
we are receiving server down notifications from pingdom every 2-3 hours and our our server is out of memory and website is taking forever to load.
when we asked to investigate the issue with our hosting provider softlayer he has replied this:
The load average has a 15 minute average of over 11.
{Jul 25 23:35 PM} [192] root@server ~ # uptime
23:35:01 up 20:15, 2 users, load average: 8.54, 10.78, 11.49
{Jul 25 23:35 PM} [193] root@server ~ #
This is due to being almost completely out of memory and dipping extensively in to swap space.
{Jul 25 23:35 PM} [194] root@server ~ # free -m
total used free shared buffers cached
Mem: 3858 3771 86 0 1 39
-/+ buffers/cache: 3730 127
Swap: 16383 14355 2028
I would get with your website's developer about optimizing the mysql queries your site is running as this is most definitely a strong contributing factor.
{Jul 25 23:35 PM} [195] root@server ~ # mysqladmin proc stat
+-------+--------------+-----------------+--------------+---------+------+-------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-------+--------------+-----------------+--------------+---------+------+-------+------------------+
| 79 | eximstats | localhost | eximstats | Sleep | 2148 | | |
| 22963 | tortix | localhost:33234 | tortix | Sleep | 114 | | |
| 28632 | leechprotect | localhost | leechprotect | Sleep | 1294 | | |
| 28752 | tortix | localhost:36675 | tortix | Sleep | 316 | | |
| 28799 | dev1979_u | localhost | dev1979_db | Sleep | 0 | | |
| 28800 | dev1979_u | localhost | dev1979_db | Sleep | 1 | | |
| 28801 | root | localhost | | Query | 0 | | show processlist |
+-------+--------------+-----------------+--------------+---------+------+-------+------------------+
Uptime: 72874 Threads: 7 Questions: 4459582 Slow queries: 60 Opens: 1231 Flush tables: 1 Open tables: 400 Queries per second avg: 61.195
There appear to be lots of blocked processes as well--
{Jul 25 23:37 PM} [197] root@server ~ # vmstat -S M 1 10
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 7 16052 83 1 39 0 0 209 112 28 22 1 0 95 3 0
0 6 16067 83 1 39 1 16 3296 16776 3085 1330 0 2 51 47 0
0 7 16073 84 1 39 0 6 xxxxxxxxxxxxxxxx - CC_FILTER 0 1 62 36 0
0 7 16075 84 1 39 1 2 xxxxxxxxxxxxxxxx - CC_FILTER 0 1 64 35 0
0 12 16078 85 1 39 1 3 2304 3244 754 445 0 1 65 34 0
0 15 16082 84 1 38 1 4 xxxxxxxxxxxxxxxx - CC_FILTER 0 2 44 54 0
0 8 16087 84 1 38 1 5 xxxxxxxxxxxxxxxx - CC_FILTER 0 1 57 42 0
0 8 16094 85 1 36 1 7 xxxxxxxxxxxxxxxx - CC_FILTER 1 1 67 31 0
0 6 16103 84 1 35 1 9 xxxxxxxxxxxxxxxx - CC_FILTER 0 1 53 46 0
1 3 16116 83 1 36 0 13 1280 13888 2709 1074 1 2 76 22 0
The script consuming the most cpu (as well as 0.6% of the memory) appears to be category.php .
{Jul 25 23:39 PM} [200] root@server ~ # ps fuxa | head -1 && ps_fuxa_sorted_by_mem | tail -20
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 30241 0.1 0.0 0 0 ? S 21:40 0:07 \_ [kworker/3:0]
tortix 25141 0.0 0.0 411004 4 ? S 23:06 0:00 \_ /var/asl/usr/sbin/tortixd
tortix 27937 0.0 0.0 410512 8 ? S 23:29 0:00 \_ /var/asl/usr/sbin/tortixd
tortix 28000 0.0 0.0 408860 0 ? S 23:30 0:00 \_ /var/asl/usr/sbin/tortixd
tortix 28056 0.0 0.0 408860 0 ? S 23:31 0:00 \_ /var/asl/usr/sbin/tortixd
ossec 3029 0.2 0.1 17656 5336 ? S 21:53 0:16 /var/ossec/bin/ossec-analysisd
tortix 27997 0.0 0.2 416916 10804 ? S 23:30 0:00 \_ /var/asl/usr/sbin/tortixd
root 2866 0.0 0.5 150736 22204 ? Ss 03:20 0:12 /usr/local/apache/bin/httpd -k start -DSSL
dev1979 28999 13.0 0.6 317184 24196 ? S 23:39 0:00 | \_ /usr/bin/php /home/dev1979/public_html/category.php
nobody 28851 0.0 0.9 151500 37996 ? S 23:38 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28126 0.0 1.0 152060 43116 ? S 23:32 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28795 0.1 1.0 151796 40428 ? S 23:36 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28800 0.0 1.0 151788 40376 ? S 23:37 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28850 0.1 1.0 151940 40652 ? S 23:38 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28857 0.1 1.0 151788 40296 ? S 23:38 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28926 0.1 1.0 151860 43224 ? S 23:39 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28703 0.1 1.1 152976 45044 ? S 23:35 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28774 0.1 1.1 151852 43644 ? S 23:36 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 28848 0.1 1.1 151860 45272 ? S 23:38 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
mysql 3662 1.7 2.4 2989488 98656 ? Sl 03:21 21:34 \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/lib/mysql/server.isorno.com.pid
{Jul 25 23:39 PM} [201] root@server ~ #
Once again, you will need to speak to your server administrator/developer about optimizing this.
Thank you for choosing SoftLayer [An IBM company]!
Pingdom report:
PingdomAlert DOWN:
isorno.com (website) is down since 2013-07-27 00:06:28.
PingdomAlert UP:
isorno.com (website) is UP again at 2013-07-27 00:21:28, after 15m of downtime.
PingdomAlert DOWN:
isorno.com (website) is down since 2013-07-27 01:46:28.
PingdomAlert UP:
isorno.com (website) is UP again at 2013-07-27 02:01:28, after 15m of downtime.
PingdomAlert DOWN:
isorno.com (website) is down since 2013-07-27 05:06:28.
PingdomAlert UP:
isorno.com (website) is UP again at 2013-07-27 05:21:28, after 15m of downtime.
PingdomAlert DOWN:
isorno.com (website) is down since 2013-07-27 06:01:28.
PingdomAlert UP:
isorno.com (website) is UP again at 2013-07-27 06:11:28, after 10m of downtime.
PingdomAlert DOWN:
isorno.com (website) is down since 2013-07-27 09:21:28.
PingdomAlert UP:
isorno.com (website) is UP again at 2013-07-27 09:31:28, after 10m of downtime.
As you can see there is some kind of attack is going on our sever. Please let us know what is causing this all.
Best Regards,
Dev