Mysql out of memory attacks

[isornoserver]

hosting provider replied this also:

Currently the PHP ising using about 17% of your memory and mysql about 17% of the CPU.

27040 isorno 20 0 306m 22m 6888 R 17.8 0.6 0:00.09 php
21115 nobody 20 0 148m 73m 2368 S 4.0 1.9 0:00.48 httpd
3842 mysql 15 -5 2673m 676m 4000 S 2.0 17.5 25:34.61 mysqld

I also see that the apache service is running on the server and that port 80 is being filtered in the firewall. If you want your website to work you need to remove port 80 from the firewall.

[~]$ nmap 5.10.88.26 -p 80 -P0
WARNING: Running Nmap setuid, as you are doing, is a major security risk.

Starting Nmap 6.00 ( http://nmap.org ) at 2013-07-30 05:33 CDT
Nmap scan report for 5.10.88.26-static.reverse.softlayer.com (5.10.88.26)
Host is up.
PORT STATE SERVICE
80/tcp filtered http

Let us know if we can be of further assistance, and thank you for choosing SoftLayer.

[isornoserver]

Hosting provider replied:

I was able to observe an event that causes elevated memory usage and was able to grab the top entry for the command:

9655 root 20 0 9495m 3.2g 148 D 8.3 84.7 0:18.15 /sbin/iptables -Z acctboth

A LSOF snapshot for this PID is as follows:

lsof -p 9655
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
iptables 9655 root cwd DIR 8,6 4096 786434 /root
iptables 9655 root rtd DIR 8,6 4096 2 /
iptables 9655 root txt REG 8,6 58872 393324 /sbin/iptables-multi-1.4.7
iptables 9655 root mem REG 8,6 22536 917544 /lib64/libdl-2.12.so
iptables 9655 root mem REG 8,6 1922152 917518 /lib64/libc-2.12.so
iptables 9655 root mem REG 8,6 598680 917566 /lib64/libm-2.12.so
iptables 9655 root mem REG 8,6 34616 917559 /lib64/libxtables.so.4.0.0-1.4.7
iptables 9655 root mem REG 8,6 28544 917625 /lib64/libip4tc.so.0.0.0-1.4.7
iptables 9655 root mem REG 8,6 156912 917511 /lib64/ld-2.12.so
iptables 9655 root 0r FIFO 0,8 0t0 4094421 pipe
iptables 9655 root 1w FIFO 0,8 0t0 4094422 pipe
iptables 9655 root 2w FIFO 0,8 0t0 4094422 pipe
iptables 9655 root 3u raw 0t0 4050571 00000000:00FF->00000000:0000 st=07

The -Z switch in the iptables command indicates that the server is zeroing the byte counter for the acctboth chain, which appears to be related to gathering bandwidth stats for cPanel/WHM. These appear be scheduled to gather stats every 2 hours.

I am curious if the bandwidth stats gathering and processing were given a different time-table (say every 12 hours) if this would change the number of alerts you receive in this regard.

[scott]

How is your nmap setuid? This doesn't sound like a clean install, that could be the root of your problem here.

[mikeshinn]

9655 root 20 0 9495m 3.2g 148 D 8.3 84.7 0:18.15 /sbin/iptables -Z acctboth

Based on the information provided, and I'm surprised your hosting company didnt know this, acctboth is a netfilter table that Cpanel sets up, and isnt part of or used by ASL. That doesnt look anything like an attack, I'm betting its a cronjob and somethings wrong with it perhaps.

Specifically, the command "/sbin/iptables -Z acctboth" is telling the kernel to zero out the counters in the netfilter tables Cpanel uses to track network statistics for accounting purposes. acctboth is a table CPanel sets up and uses. It doesnt otherwise exist, or get used by anything else, just CPanel. So, assuming the information your hosting provider provided is complete and correct, it looks like either CPanel is doing this, or some other product is. Either way, it appears to be using up a considerable amount of memory.

You'll want to have CPanel look into this bug. That table, and command, have nothing to do with ASL. Sorry we cant help you further, but this command isnt called by ASL nor to we setup or use that netfilter table.

[isornoserver]

hosting admin replied this:

The server configuration appears to be fairly well optimized; at this point your best gain would be adding additional RAM to allow mysql to make use of it as mysql appears to be your biggest consumer of memory at this point. We'd recommend going to at least 4 GB if not 8GB; the system will support a max of 32GB.

Watching your vmstat 5 output, the server is doing quite a bit of swapping in and out which also confirms this.

Ideally you want the si/so to stay at 0.

[root@server ~]# vmstat 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 2 984696 147444 20148 2468316 10 77 203 116 3 7 4 0 94 2 0
0 1 984688 285544 17112 2358828 3 0 3xxxxxxxxxxxxxxxx - CC_FILTER 2 1 74 23 0
1 2 984660 261532 17532 2382160 19 0 xxxxxxxxxxxxxxxx - CC_FILTER 2 0 83 15 0
0 1 984640 233004 17736 2398708 2 0 1662 6275 615 805 1 0 81 18 0
0 1 984628 225508 18480 2417708 14 0 xxxxxxxxxxxxxxxx - CC_FILTER 4 0 81 15 0
0 3 984620 175788 18908 2442680 6 0 xxxxxxxxxxxxxxxx - CC_FILTER 3 1 79 18 0
1 2 984620 141112 19524 2470792 0 0 xxxxxxxxxxxxxxxx - CC_FILTER 2 1 80 18 0
0 1 984612 150864 20128 2490652 0 0 xxxxxxxxxxxxxxxx - CC_FILTER 2 0 81 16 0
2 1 984668 137340 18984 2484440 11 12 1xxxxxxxxxxxxxxxx - CC_FILTER 3 1 73 24 0
0 1 984656 166108 18552 2476480 61 0 xxxxxxxxxxxxxxxx - CC_FILTER 3 0 80 17 0
0 1 984656 147056 18796 2494640 0 0 xxxxxxxxxxxxxxxx - CC_FILTER 2 0 83 15 0
1 1 984652 127320 19456 2514076 0 0 xxxxxxxxxxxxxxxx - CC_FILTER 2 0 83 15 0
0 6 984440 162352 17388 2480984 102 2 2399 7462 695 849 1 0 81 18 0
0 1 984424 133840 17636 2504752 641 0 2926 9460 809 1073 1 0 83 15 0
2 6 984424 123676 15768 2483552 650 0 xxxxxxxxxxxxxxxx - CC_FILTER 3 1 73 24 0
0 6 984448 274228 14336 2359712 483 7 1xxxxxxxxxxxxxxxx - CC_FILTER 4 1 68 28 0
1 2 984436 233824 14628 2383700 787 0 xxxxxxxxxxxxxxxx - CC_FILTER 2 0 76 22 0
0 2 984432 218264 15088 2407552 449 0 xxxxxxxxxxxxxxxx - CC_FILTER 2 0 79 19 0
0 11 984432 177268 15516 2428924 391 0 xxxxxxxxxxxxxxxx - CC_FILTER 4 1 75 21 0
0 4 984424 149700 15908 2452368 213 0 xxxxxxxxxxxxxxxx - CC_FILTER 3 1 67 29 0
0 4 984380 107680 17016 2478244 43 0 xxxxxxxxxxxxxxxx - CC_FILTER 3 1 75 22 0
0 1 984332 173928 16524 2447124 62 2 xxxxxxxxxxxxxxxx - CC_FILTER 1 0 81 18 0
0 3 981792 142980 16812 2471124 593 0 xxxxxxxxxxxxxxxx - CC_FILTER 2 1 78 19 0
0 3 976076 125900 17168 2484164 1037 0 xxxxxxxxxxxxxxxx - CC_FILTER 1 1 82 17 0
0 2 974740 134596 16824 2469868 237 2 xxxxxxxxxxxxxxxx - CC_FILTER 1 1 78 21 0
0 2 969732 122800 17084 2483144 142 0 270 8165 937 1000 1 1 84 14 0
0 1 969736 142912 16560 2462016 0 1 77 7080 769 846 1 0 88 11 0
0 2 969732 134632 16796 2470556 0 0 77 6898 781 882 1 1 85 14 0
1 2 969724 117256 17028 2483180 14 0 85 7941 734 874 1 1 84 15 0
1 2 969708 115536 17272 2487772 14 0 87 5629 644 823 1 0 85 14 0
0 1 969708 164012 16700 2440564 58 1 317 6035 1749 1888 2 1 78 19 0
0 2 969708 149948 16948 2453428 0 0 70 8195 505 664 0 0 86 13 0


As he is asking to upgrade RAM but our website only require 512MB to run smoothly and we already have 4GB RAM dedicated to single website do you thing upgrading RAM will solve the issue?

Please advise. Thanks

[mikeshinn]

Well, given that this isnt related to ASL, I'm going to move this to the General Help forums.

My two cents: Didnt they tell you earlier that CPanels iptables command was using up all your memory? If thats true, adding more memory isnt likely to solve your problem. iptables shouldnt be using that much memory, so I wouldnt be surprised if this happens again even if you add more memory. Adding more memory isnt going to fix whatever is wrong with CPanel.

With that said, if you are paying your hosting company to do this kind of analysis on your system, then you should probably go with what they say. If you dont trust them to do that for you, then I'd hire an outside expert to look at your system to see whats going on. (And maybe drop your current hosting company if they are supposed to do this for you)

I wish I could give you a more direct answer about whats causing your OOM errors, but I'd have to collect data on the system and look at your logs to really sort this out (and I've already got a day job and an 80 hour work week! *grin*).

Theres lots of smart people on these forums that provide that service, and I'm sure they would be happy to help you for a small fee.

[isornoserver]

I have asked my hosting provider to look for detailed information and here is what they replied:

Here's the current output from top. At the moment, there appear to be no memory issues at all, you have 1.3G free and plenty of swap free, along with .8G of cache open and no I/O issues, but you can see mysqld is the top user of memory:


top - 10:47:29 up 4 days, 2:18, 1 user, load average: 0.04, 0.21, 0.56
Tasks: 158 total, 2 running, 155 sleeping, 0 stopped, 1 zombie
Cpu(s): 0.9%us, 0.2%sy, 0.0%ni, 98.8%id, 0.2%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 3950936k total, 2642100k used, 1308836k free, 15216k buffers
Swap: 16777212k total, 550476k used, 16226736k free, 811676k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13842 mysql 15 -5 2586m 1.2g 5228 S 0.7 32.8 0:54.77 mysqld
13945 nobody 20 0 148m 72m 2296 S 0.0 1.9 0:01.65 httpd
13948 nobody 20 0 148m 72m 2352 S 0.0 1.9 0:00.77 httpd
13957 nobody 20 0 148m 72m 2232 S 0.0 1.9 0:01.68 httpd
15926 nobody 20 0 148m 72m 2204 S 0.0 1.9 0:01.01 httpd
13947 nobody 20 0 148m 72m 2296 S 0.0 1.9 0:00.60 httpd
13958 nobody 20 0 148m 72m 2352 S 0.3 1.9 0:00.61 httpd
13946 nobody 20 0 148m 72m 2292 S 0.0 1.9 0:00.87 httpd
13973 nobody 20 0 148m 71m 2280 S 0.0 1.9 0:00.65 httpd
16112 nobody 20 0 148m 71m 2176 S 0.0 1.9 0:00.03 httpd
20165 root 20 0 147m 71m 3056 S 0.0 1.9 0:12.83 httpd
16137 nobody 20 0 147m 71m 2108 S 0.0 1.8 0:00.00 httpd
10022 tortix 30 10 499m 45m 6268 S 0.0 1.2 0:06.73 tortixd
5174 tortix 30 10 499m 44m 5752 S 0.0 1.2 0:09.83 tortixd
5920 tortix 30 10 499m 44m 5996 S 0.0 1.2 0:08.37 tortixd
8031 tortix 30 10 497m 43m 4616 S 0.0 1.1 0:06.99 tortixd

I will have some other notes here shortly.

Next Reply:

Ok, I can't say that it is mysql that is responsible for the memory usage, just that right now it is responsible for the most memory usage right now. During your previous memory spikes, a variety of processes were singled out by the kernel for termination:

root@server log]# grep oom messages
Jul 28 08:29:49 server kernel: udevd (104): /proc/104/oom_adj is deprecated, please use /proc/104/oom_score_adj instead.
Jul 28 14:02:51 server kernel: ossec-logcollec invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jul 28 14:02:51 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 28 14:02:51 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 29 11:06:08 server kernel: processes invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jul 29 11:06:08 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 29 11:06:08 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 29 15:02:40 server kernel: httpd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jul 29 15:02:40 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 29 15:02:40 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 29 18:04:05 server kernel: iptables invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jul 29 18:04:05 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 29 18:04:05 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 29 20:08:47 server kernel: ossec-logcollec invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jul 29 20:08:48 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 29 20:08:48 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 30 12:07:32 server kernel: iptables invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jul 30 12:07:33 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 30 12:07:33 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 30 18:09:59 server kernel: iptables invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jul 30 18:09:59 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 30 18:10:01 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 30 22:11:47 server kernel: iptables invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jul 30 22:11:47 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 30 22:11:48 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 31 11:04:50 server kernel: iptables invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jul 31 11:04:50 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 31 11:04:50 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 31 15:06:18 server kernel: iptables invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jul 31 15:06:18 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 31 15:06:19 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 31 22:09:14 server kernel: processes invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jul 31 22:09:14 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 31 22:09:15 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 31 23:23:24 server kernel: /usr/local/cpan invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jul 31 23:23:24 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 31 23:23:24 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Aug 1 09:21:38 server kernel: dovecot invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Aug 1 09:21:39 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Aug 1 09:21:40 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name

dovecot, iptables, cpan, httpd, tis is quite an interesting list, but it doesn't always mean that these were the processes responsible for causing the memory to be used, just that they were using memory at the time and had low oom_score and were subject to being killed by the kernel to free up memory to stay running.

Unfortunately the logging of the server isn't granular enough to tell me what exactly was running at these times. However, I've attached a zipfile containing three sets of graphs for the past three days showing system activity. On the 30th you can see where your system was doing pretty well, your memory usage will show near 100% but this is fine, since some of it is composed of buffers and cache, but around 1800 server time this vanishes and the cache is used up, and later the swap fills up as well and you get a series of oom_kill messages in the logs at both points. In fact, if you look at the oom_kill entries, you can see memory usage increases and load spikes along each of these log entries, so something is happening at these times, but again, the logs only shows system status and not what processes are actually running.

There are no hardware errors associated with these events, the server itself appears to be fine. You may want to check and intervals for these events and make sure that you aren't running backups or something at these times, or that you don't have regular cron jobs or something interfering with server operation.


They have provided me log file as well but i can't find file upload option here so please let me know how can i send you log file. thanks

[isornoserver]

it says file is too big my file size is 4MB

can you provide me email where i can send this zip file. Thanks

[isornoserver]

I'm happy to pay small fee if someone can really solve my server problem. please let me know.

[isornoserver]

OOM log messages:

root@server ~]# zgrep oom /var/log/messages*
Jul 28 08:29:49 server kernel: udevd (104): /proc/104/oom_adj is deprecated, please use /proc/104/oom_score_adj instead.
Jul 28 14:02:51 server kernel: ossec-logcollec invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jul 28 14:02:51 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 28 14:02:51 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 29 11:06:08 server kernel: processes invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jul 29 11:06:08 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 29 11:06:08 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 29 15:02:40 server kernel: httpd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jul 29 15:02:40 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 29 15:02:40 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 29 18:04:05 server kernel: iptables invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jul 29 18:04:05 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 29 18:04:05 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 29 20:08:47 server kernel: ossec-logcollec invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jul 29 20:08:48 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 29 20:08:48 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 30 12:07:32 server kernel: iptables invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jul 30 12:07:33 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 30 12:07:33 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 30 18:09:59 server kernel: iptables invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jul 30 18:09:59 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 30 18:10:01 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 30 22:11:47 server kernel: iptables invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jul 30 22:11:47 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 30 22:11:48 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 31 11:04:50 server kernel: iptables invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jul 31 11:04:50 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 31 11:04:50 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 31 15:06:18 server kernel: iptables invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jul 31 15:06:18 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 31 15:06:19 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 31 22:09:14 server kernel: processes invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jul 31 22:09:14 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 31 22:09:15 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 31 23:23:24 server kernel: /usr/local/cpan invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jul 31 23:23:24 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 31 23:23:24 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Aug 1 09:21:38 server kernel: dovecot invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Aug 1 09:21:39 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Aug 1 09:21:40 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Aug 1 15:07:47 server kernel: mysqld invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Aug 1 15:07:47 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Aug 1 15:07:47 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Aug 1 17:04:53 server kernel: iptables invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Aug 1 17:04:53 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Aug 1 17:04:53 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Aug 1 19:07:19 server kernel: spooler invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Aug 1 19:07:19 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Aug 1 19:07:19 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name

And:

[root@server ~]# grep oom /var/log/messages*
Jul 28 08:29:49 server kernel: udevd (104): /proc/104/oom_adj is deprecated, please use /proc/104/oom_score_adj instead.
Jul 28 14:02:51 server kernel: ossec-logcollec invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jul 28 14:02:51 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 28 14:02:51 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 29 11:06:08 server kernel: processes invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jul 29 11:06:08 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 29 11:06:08 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 29 15:02:40 server kernel: httpd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jul 29 15:02:40 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 29 15:02:40 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 29 18:04:05 server kernel: iptables invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jul 29 18:04:05 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 29 18:04:05 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 29 20:08:47 server kernel: ossec-logcollec invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jul 29 20:08:48 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 29 20:08:48 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 30 12:07:32 server kernel: iptables invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jul 30 12:07:33 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 30 12:07:33 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 30 18:09:59 server kernel: iptables invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jul 30 18:09:59 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 30 18:10:01 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 30 22:11:47 server kernel: iptables invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jul 30 22:11:47 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 30 22:11:48 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 31 11:04:50 server kernel: iptables invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jul 31 11:04:50 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 31 11:04:50 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 31 15:06:18 server kernel: iptables invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jul 31 15:06:18 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 31 15:06:19 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 31 22:09:14 server kernel: processes invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jul 31 22:09:14 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 31 22:09:15 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 31 23:23:24 server kernel: /usr/local/cpan invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jul 31 23:23:24 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 31 23:23:24 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Aug 1 09:21:38 server kernel: dovecot invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Aug 1 09:21:39 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Aug 1 09:21:40 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Aug 1 15:07:47 server kernel: mysqld invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Aug 1 15:07:47 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Aug 1 15:07:47 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Aug 1 17:04:53 server kernel: iptables invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Aug 1 17:04:53 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Aug 1 17:04:53 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Aug 1 19:07:19 server kernel: spooler invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Aug 1 19:07:19 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Aug 1 19:07:19 server kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
[root@server ~]#

You will need to read through the file using the command line based tools like nano or using less to view the output and then searching for oom so you can view the entire logged issue:

Jul 28 14:02:51 server kernel: ossec-logcollec invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jul 28 14:02:51 server kernel: ossec-logcollec cpuset=/ mems_allowed=0
Jul 28 14:02:51 server kernel: Pid: 889, comm: ossec-logcollec Tainted: G C 3.2.48-54.art.x86_64 #1
Jul 28 14:02:51 server kernel: Call Trace:
Jul 28 14:02:51 server kernel: [<ffffffff810e111a>] dump_header+0x7f/0x1ab
Jul 28 14:02:51 server kernel: [<ffffffff81096067>] ? cpuset_mems_allowed_intersects+0x21/0x23
Jul 28 14:02:51 server kernel: [<ffffffff81217b15>] ? security_real_capable_noaudit+0x3e/0x6a
Jul 28 14:02:51 server kernel: [<ffffffff810e1589>] oom_kill_process+0x50/0x27c
Jul 28 14:02:51 server kernel: [<ffffffff810e1aad>] out_of_memory+0x2f8/0x365
Jul 28 14:02:51 server kernel: [<ffffffff810e5fa8>] __alloc_pages_nodemask+0x5fd/0x772
Jul 28 14:02:51 server kernel: [<ffffffff81114297>] alloc_pages_current+0xc7/0xe8
Jul 28 14:02:51 server kernel: [<ffffffff810ded97>] __page_cache_alloc+0x8c/0x99
Jul 28 14:02:51 server kernel: [<ffffffff810e0698>] filemap_fault+0x25b/0x353
Jul 28 14:02:51 server kernel: [<ffffffff810fea3b>] __do_fault+0xc9/0x434
Jul 28 14:02:51 server kernel: [<ffffffff810ffb59>] handle_pte_fault+0x2c9/0x8ed
Jul 28 14:02:51 server kernel: [<ffffffff810ffb59>] ? handle_pte_fault+0x2c9/0x8ed
Jul 28 14:02:51 server kernel: [<ffffffff8104f905>] ? sys_time+0x1a/0x4a
Jul 28 14:02:51 server kernel: [<ffffffff811003f3>] handle_mm_fault+0x276/0x290
Jul 28 14:02:51 server kernel: [<ffffffff814c82cc>] do_page_fault+0x3eb/0x432
Jul 28 14:02:51 server kernel: [<ffffffff8102a9f3>] ? bad_area_nosemaphore+0x13/0x15
Jul 28 14:02:51 server kernel: [<ffffffff814c8123>] ? do_page_fault+0x242/0x432
Jul 28 14:02:51 server kernel: [<ffffffff814cba69>] ? pax_enter_kernel_user+0xf9/0x110
Jul 28 14:02:51 server kernel: [<ffffffff814c5abb>] page_fault+0x3b/0x40
Jul 28 14:02:51 server kernel: Mem-Info:
Jul 28 14:02:51 server kernel: Node 0 DMA per-cpu:
Jul 28 14:02:51 server kernel: CPU 0: hi: 0, btch: 1 usd: 0
Jul 28 14:02:51 server kernel: CPU 1: hi: 0, btch: 1 usd: 0
Jul 28 14:02:51 server kernel: CPU 2: hi: 0, btch: 1 usd: 0
Jul 28 14:02:51 server kernel: CPU 3: hi: 0, btch: 1 usd: 0
Jul 28 14:02:51 server kernel: CPU 4: hi: 0, btch: 1 usd: 0
Jul 28 14:02:51 server kernel: CPU 5: hi: 0, btch: 1 usd: 0
Jul 28 14:02:51 server kernel: CPU 6: hi: 0, btch: 1 usd: 0
Jul 28 14:02:51 server kernel: CPU 7: hi: 0, btch: 1 usd: 0
Jul 28 14:02:51 server kernel: Node 0 DMA32 per-cpu:

[isornoserver]

ASL log:

1 August
23:25:58 server 12 5108 server kernel: Out of memory: Kill process 11342 (iptables) score 824 or sacrifice chil
19:07:25 server 12 5108 server kernel: Out of memory: Kill process 19621 (iptables) score 835 or sacrifice chil
17:05:05 server 12 5108 server kernel: Out of memory: Kill process 10409 (iptables) score 837 or sacrifice chil
15:07:53 server 12 5108 server kernel: Out of memory: Kill process 3932 (iptables) score 831 or sacrifice chil
10:12:31 server 12 50120 MySQL log: 130801 10:12:22 InnoDB: Shutdown completed; log sequence number 1409245677
10:12:31 server 12 50120 MySQL log: 130801 10:12:22 [Note] /usr/sbin/mysqld: Shutdown complet
10:12:31 server 7 553 File `/usr/local/lib/mysql.sock` was deleted. Unable to retrieve checksum
09:54:42 server 9 330045
66.135.38.216 Atomicorp.com WAF Rules: Suspicious Unusual User Agent (pycurl). Disable this rule if you use pycurl.
09:21:48 server 12 5108 server kernel: Out of memory: Kill process 7355 (iptables) score 836 or sacrifice chil
06:10:20 server 7 300071
122.177.147.187 Atomicorp.com WAF AntiSpam Rules: Possible SEO or spamware content
05:56:30 server 7 552 Integrity checksum changed for: `/etc/rc.d/init.d/ipaliasesSize changed from `8540` to `0What changed1,253d< #!/bin/s< < # ipaliases Bring up/down ipaliase< < # chkconfig: 2345 11 9< # description: Activates/Deactivates all ip aliase< # probe: fals<< # Source function library< if [ -f /etc/init.d/functions ]; the< . /etc/init.d/function< elif [ -f /etc/rc.d/init.d/functions ]; the< . /etc/rc.d/init.d/function< els< echo "Could not find functions file, your system may be broken< exit
05:56:30 server 7 553 File `/etc/rc.d/rc3.d/S11ipaliases` was deleted. Unable to retrieve checksum

31 July
23:58:06 server 8 30302 server kernel: grsec: From 208.167.230.35: Segmentation fault occurred at (nil) in /usr/bin/php[php:10699] uid/euid:500/500 gid/egid:501/501, parent /usr/local/apache/bin/httpd[httpd:3185] uid/euid:99/99 gid/egid:99/9
23:23:41 server 12 5108 server kernel: Out of memory: Kill process 2446 (iptables) score 846 or sacrifice chil
08:01:42 server 7 552 Integrity checksum changed for: `/etc/sysconfig/rkhunterSize changed from `338` to `0
07:49:48 server 7 551 Integrity checksum changed for: `/etc/shadow-What changed1c< root:$1$PTWuaSoZ$tCrGhC5MzCC1U3aN1itAa/0:99999:7::--> root:$1$Jx4gbrbQ$7qiLXqDiTMnhKgjUWCgSR10:99999:7::
05:31:10 server 7 552 Integrity checksum changed for: `/etc/asl/geo-blacklistSize changed from `45` to `48What changed15a1> u
00:58:53 server 7 550 Integrity checksum changed for: `/etc/shadow
00:57:28 server 7 550 Integrity checksum changed for: `/etc/passwd.nouids.cache
00:57:28 server 7 550 Integrity checksum changed for: `/etc/passwd.cache


as you can see ASL is displaying Out of memory warnings, and now i have already posted oom error log also could you please let me know why our server is getting out of memory who is the culprit?

[isornoserver]

No one is able to help here so i have no choice but to uninstall ASL completely as this all has started just after installing ASL. Very disappointed.

[hostingg]

what makes you think asl has anything to do with this? it looks like your hosting company said this was being caused by cpanel iptables. have you even contacted cpanel support?

[mikeshinn]

No one is able to help here so i have no choice but to uninstall ASL completely as this all has started just after installing ASL. Very disappointed.

I cant find that you've opened a case with our support team. If you have an issue with our products, please open a case so our support team can help you. The forums are not the supported method for getting help. Please use one of the methods described below:

https://www.atomicorp.com/wiki/index.ph ... rt_Methods

With that said, OOM errors are your kernels way of trying to save your system from completely running out of memory. ASL does not cause these, your operating system is doing this because something is using up all your memory. Your hosting company has reported that you had a process, iptables -Z acctboth, that was using up all your memory. That process is part of a program called CPanel, which is the control panel you are using on your system. That program is using up all your memory.

CPanel is not part of ASL, and is not supported by us, its supported by CPanel:

http://www.cpanel.net

This best way to find out why CPanel is using up all your memory is to contact CPanel. Please let us all know what CPanel has to say about why CPanel's command "iptables -Z acctboth" is using up all your memory. It should not do this, but its CPanels command so perhaps they have a reason for why they need to use up all your memory. I suspect that this is a bug in CPanel, as opposed to expected behavior, so we'd all be very interested to know what this bug in CPanel is, and how CPanel recommends you remedy it.

Please let us know what CPanel says. Its vital we know why CPanel is using up all the memory on your system. Once we know that, we can help you with your issue.

[isornoserver]

Thanks for your reply. i have reinstalled whole system again and its been 2 days i'm not facing server down issue but there is strange think going on in the system.

i have enabled php check in the system and disabled all functions except curl_exec as its required to run PayPal module but i have noticed every 30-45min vulnerability details windows shows warnings for:

High Risk: Allow URL Fopen is enabled. This allows an attacker to remotely include files into PHP scripts through urls. Read More...
High Risk: Allow URL Include is enabled. This allows an attacker to remotely include files into PHP scripts through urls. Read More...
High Risk: PHP function curl_exec() allows an attacker to execute shell commands through php. Read More...
High Risk: SSH No Administrative users are defined: Administrative users are the users that maintain this system, that should su or sudo to root. This test verifies that administrative users are defined. It is not recommended to manage the system by directly logging in as root. Read More...
High Risk: SSH setting: The system allows remote root logins. Read More...
Moderate Risk: display_errors is enabled. This exposes error messages to external attackers. Read More...
Moderate Risk: psmon is not installed. Psmon is the daemon that monitors critical system, and ASL services for downtime. Read More...
Moderate Risk: psmon is disabled. Read More...

and when i update ASL configuration

High Risk: PHP function curl_exec() allows an attacker to execute shell commands through php. Read More...
High Risk: SSH No Administrative users are defined: Administrative users are the users that maintain this system, that should su or sudo to root. This test verifies that administrative users are defined. It is not recommended to manage the system by directly logging in as root. Read More...
High Risk: SSH setting: The system allows remote root logins. Read More...
Moderate Risk: psmon is not installed. Psmon is the daemon that monitors critical system, and ASL services for downtime. Read More...
Moderate Risk: psmon is disabled. Read More...

but it takes about 30-45min after update and again the same warning are shown in vulnerability details windows:
High Risk: Allow URL Fopen is enabled. This allows an attacker to remotely include files into PHP scripts through urls. Read More...
High Risk: Allow URL Include is enabled. This allows an attacker to remotely include files into PHP scripts through urls. Read More...
High Risk: PHP function curl_exec() allows an attacker to execute shell commands through php. Read More...
High Risk: SSH No Administrative users are defined: Administrative users are the users that maintain this system, that should su or sudo to root. This test verifies that administrative users are defined. It is not recommended to manage the system by directly logging in as root. Read More...
High Risk: SSH setting: The system allows remote root logins. Read More...
Moderate Risk: display_errors is enabled. This exposes error messages to external attackers. Read More...
Moderate Risk: psmon is not installed. Psmon is the daemon that monitors critical system, and ASL services for downtime. Read More...
Moderate Risk: psmon is disabled. Read More...

i have updated ASL configuration many times but its not solving this problem. so could you please tell me what is going on?

Best Regards,
Dev