Page 1 of 1
swap is killing servers instead of saving them
Posted: Fri Oct 21, 2011 10:41 am
by nobody
Ok. Over the last few months I am thinking of disabling swap on my servers.
It has turned out to be a real pain in the *ss.
I see servers without having memory pressure since 20-50% of their memory is shown as cached and not us active and I have mysql going to swap and writing even 500mbyte taking the disk into death.
I have also found many people which have the same issues.
Offcourse some ammount of swap is good for emergency but every crash I have suffered over the last 3 years is due to swap.
A nice article explaining the numa arch.
Code: Select all
http://jcole.us/blog/archives/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/
Re: swap is killing servers instead of saving them
Posted: Tue Oct 25, 2011 6:45 pm
by nobody
Since today I have disabled swap. I will try to push as much as I can the server and I'll let you know !
Re: swap is killing servers instead of saving them
Posted: Tue Oct 25, 2011 7:13 pm
by scott
Is this on a x86_64 or i386 box? What does your /etc/my.cnf look like?
Oh and what kernel are you on?
Re: swap is killing servers instead of saving them
Posted: Tue Oct 25, 2011 7:22 pm
by nobody
scott wrote:Is this on a x86_64 or i386 box? What does your /etc/my.cnf look like?
Oh and what kernel are you on?
Thanks for your reply Scott !
This is a x86_64.
Kernel version is
Code: Select all
Linux server.hostname.com 2.6.32.43-6.art.x86_64 #1 SMP Thu Jul 14 14:14:48 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
my.cnf
Code: Select all
[mysqld]
local-infile=0
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
old_passwords=1
thread_concurrency=4
max_connections = 150
max_user_connections = 150
key_buffer = 150M
myisam_sort_buffer_size = 64M
join_buffer_size = 2M
read_buffer_size = 1M
sort_buffer_size = 1M
table_cache = 1024
thread_cache_size = 128
interactive_timeout = 25
wait_timeout = 1800
connect_timeout = 10
max_allowed_packet = 1M
max_connect_errors = 1000
query_cache_limit = 1M
query_cache_size = 32M
query_cache_type = 1
tmp_table_size = 16M
max_heap_table_size = 16M
# symbolic-links=0
Re: swap is killing servers instead of saving them
Posted: Wed Oct 26, 2011 10:20 am
by scott
Ok so, can you try this to see if it helps:
Install numactl:
yum install numactl
back up /usr/bin/mysqld_safe
Find this line (this is if you're using mysql 5.5) in /usr/bin/mysqld_safe
cmd="`mysqld_ld_preload_text`$NOHUP_NICENESS"
add after that line (dont replace it!):
cmd="/usr/bin/numactl --interleave all $cmd"
and restart mysqld
Re: swap is killing servers instead of saving them
Posted: Thu Oct 27, 2011 8:55 am
by nobody
Well I can try that.
But its not only mysql. I see 2gb cache and 380mb on swap. This 380mb are from all around services. Apache, mysql, qmail, clamav etc ...
So even if I solve the mysql problem I would still pretty much have the same issue.