swap is killing servers instead of saving them

General Discussion of atomic repo and development projects.

Ask for help here with anything else not covered by other forums.
nobody
Forum Regular
Forum Regular
Posts: 349
Joined: Sun Mar 29, 2009 6:52 pm

swap is killing servers instead of saving them

Unread post by nobody »

Ok. Over the last few months I am thinking of disabling swap on my servers.
It has turned out to be a real pain in the *ss.

I see servers without having memory pressure since 20-50% of their memory is shown as cached and not us active and I have mysql going to swap and writing even 500mbyte taking the disk into death.
I have also found many people which have the same issues.
Offcourse some ammount of swap is good for emergency but every crash I have suffered over the last 3 years is due to swap.

A nice article explaining the numa arch.

Code: Select all

http://jcole.us/blog/archives/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/
Hello IT.
Phone : Blah Blah ....
Have you tried turning it on and off again ?
Phone : Blah Blah ....
....
I'm sorry, are you from the Past ?!
http://www.youtube.com/watch?v=-E4fm4Wqego
nobody
Forum Regular
Forum Regular
Posts: 349
Joined: Sun Mar 29, 2009 6:52 pm

Re: swap is killing servers instead of saving them

Unread post by nobody »

Since today I have disabled swap. I will try to push as much as I can the server and I'll let you know !
Hello IT.
Phone : Blah Blah ....
Have you tried turning it on and off again ?
Phone : Blah Blah ....
....
I'm sorry, are you from the Past ?!
http://www.youtube.com/watch?v=-E4fm4Wqego
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8355
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Re: swap is killing servers instead of saving them

Unread post by scott »

Is this on a x86_64 or i386 box? What does your /etc/my.cnf look like?

Oh and what kernel are you on?
nobody
Forum Regular
Forum Regular
Posts: 349
Joined: Sun Mar 29, 2009 6:52 pm

Re: swap is killing servers instead of saving them

Unread post by nobody »

scott wrote:Is this on a x86_64 or i386 box? What does your /etc/my.cnf look like?

Oh and what kernel are you on?
Thanks for your reply Scott !

This is a x86_64.

Kernel version is

Code: Select all

Linux server.hostname.com 2.6.32.43-6.art.x86_64 #1 SMP Thu Jul 14 14:14:48 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
my.cnf

Code: Select all

[mysqld]
local-infile=0
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
old_passwords=1

thread_concurrency=4
max_connections = 150
max_user_connections = 150
key_buffer = 150M
myisam_sort_buffer_size = 64M
join_buffer_size = 2M
read_buffer_size = 1M
sort_buffer_size = 1M
table_cache = 1024
thread_cache_size = 128
interactive_timeout = 25
wait_timeout = 1800
connect_timeout = 10
max_allowed_packet = 1M
max_connect_errors = 1000
query_cache_limit = 1M
query_cache_size = 32M
query_cache_type = 1
tmp_table_size = 16M
max_heap_table_size = 16M

# symbolic-links=0
Hello IT.
Phone : Blah Blah ....
Have you tried turning it on and off again ?
Phone : Blah Blah ....
....
I'm sorry, are you from the Past ?!
http://www.youtube.com/watch?v=-E4fm4Wqego
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8355
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Re: swap is killing servers instead of saving them

Unread post by scott »

Ok so, can you try this to see if it helps:

Install numactl:
yum install numactl

back up /usr/bin/mysqld_safe

Find this line (this is if you're using mysql 5.5) in /usr/bin/mysqld_safe

cmd="`mysqld_ld_preload_text`$NOHUP_NICENESS"

add after that line (dont replace it!):

cmd="/usr/bin/numactl --interleave all $cmd"

and restart mysqld
nobody
Forum Regular
Forum Regular
Posts: 349
Joined: Sun Mar 29, 2009 6:52 pm

Re: swap is killing servers instead of saving them

Unread post by nobody »

Well I can try that.
But its not only mysql. I see 2gb cache and 380mb on swap. This 380mb are from all around services. Apache, mysql, qmail, clamav etc ...

So even if I solve the mysql problem I would still pretty much have the same issue.
Hello IT.
Phone : Blah Blah ....
Have you tried turning it on and off again ?
Phone : Blah Blah ....
....
I'm sorry, are you from the Past ?!
http://www.youtube.com/watch?v=-E4fm4Wqego
Post Reply