Server Hang - kswapd process the cause?

[philb]

when monitoring top, the server load is generally httpd commands. At the time of writing, the server load is low and all is OK, and we have:

[Snip...]

and once we had about 60 of them and because it had hung, we could not even kill them.

How did you try to kill them? Kill -9 should always work because you are telling the kernel to cut them off at the knees. Killall uses sigterm (which is kill -15) and that assumes that whatever you are trying to kill is listening. If that doesn't work, then kill -9 pid will always work. If it doesn't your kernel is broken and you really do have to reboot.

The difference between kill -9 and kill -15

kill -15 sends a signal to the process that we would like it to stop and then we wait for it go get around to it. This allows it to finish it's writes, etc. and exit cleanly. Eventually. Maybe.

kill -9 sends a signal to the kernel to kill with extreme prejudice the pid you sent it. It pretty much always works, but the possibility of data loss exists because you're not playing nice with the process.

[coolemail]

It was kill -9 that did not work because something was already guzzling the CPU (we think kswapd). Mike has helped identify that it appears a mysql database is broken. I think it might be the whole ASL because we are getting emails from IPs which have been blacklisted and because security events are now not loading.

We need to address that and then see whether it improves.

[philb]

It was kill -9 that did not work because something was already guzzling the CPU (we think kswapd). Mike has helped identify that it appears a mysql database is broken. I think it might be the whole ASL because we are getting emails from IPs which have been blacklisted and because security events are now not loading.

We need to address that and then see whether it improves.

Yes, once you start into kswapd, you are running about 1000 times slower than if you are in memory.

You should probably add ram. Sometimes just bumping into the ram limit causes an avalanche once swapping starts. 2G these days is pretty small and even server memory is cheap.

[scott]

If you've been using mysqltuner (which is awesome, it just doesn't know anything about the resources you have) you may have overtuned my.cnf. I've seen it get overdone to the point of causing problems like this.