Server Crash/Lockup Analysis Help

[damon]

I have a 1and1 server that has been running well suddenly went down, couldn't be reached via ssh, like it didn't exist.

Went to admin.1and1.com and rebooted in normal mode and everything came back up.

Looking at the messages log there were 500 lines of unusual messages right before the crash.

Could this be a physical problem in the server or does it look like a Centos or App problem? This isn't the first time this has happened, but haven't seen any pattern to it.

Any insight into where to look for the problem would be great.

Here is a text file with the messages.
http://www.memwow.com/messages-log-crash.txt

This server is using the Art Centos and ASL repos. Here is some other info.

# cat /proc/version
Linux version 2.6.27.7-9.art.x86_64 (mockbuild@archelon.atomicorp.com) (gcc version 4.3.2 20081105 (Red Hat 4.3.2-7) (GCC) ) #1 SMP Fri Dec 19 11:42:37 EST 2008

]# cat /etc/redhat-release
CentOS release 5.3 (Final)

# cat /proc/meminfo
MemTotal: 2025144 kB
MemFree: 66916 kB
Buffers: 84836 kB
Cached: 699400 kB
SwapCached: 0 kB
Active: 1363304 kB
Inactive: 357620 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 1836 kB
Writeback: 0 kB
AnonPages: 936636 kB
Mapped: 50024 kB
Slab: 174308 kB
SReclaimable: 161812 kB
SUnreclaim: 12496 kB
PageTables: 23712 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 1012572 kB
Committed_AS: 1486308 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 13020 kB
VmallocChunk: 34359725039 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 3456 kB
DirectMap2M: 2060288 kB

[scott]

yeah you're OOM'ing on httpd, looks like you dont have swap set up. Was that an AOOI re-image?

[damon]

yeah you're OOM'ing on httpd, looks like you dont have swap set up. Was that an AOOI re-image?

Yes, this is AOOI.

[damon]

Darn, no swap makes sense. I was all proud that the server wasn't getting into swap and it turns out there wasn't any!

Code: Select all

# free -m
             total       used       free     shared    buffers     cached
Mem:          1977       1934         43          0         71        694
-/+ buffers/cache:       1168        809
Swap:            0          0          0

Should I add the swap with the commands here?
http://www.cyberciti.biz/faq/linux-add- ... ile-howto/

[scott]

yup, looks like you dont have a swap partition. I'll bet it already exists though, take a look with:

fdisk -l

[damon]

Code: Select all

# fdisk -l

Disk /dev/sda: 250.0 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          33      265041   fd  Linux raid autodetect
/dev/sda2             124         367     1959930   82  Linux swap / Solaris
/dev/sda3             368       30401   241248105   fd  Linux raid autodetect

Disk /dev/sdb: 250.0 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          33      265041   fd  Linux raid autodetect
/dev/sdb2             124         367     1959930   82  Linux swap / Solaris
/dev/sdb3             368       30401   241248105   fd  Linux raid autodetect

Disk /dev/md0: 247.0 GB, 247037952000 bytes
2 heads, 4 sectors/track, 60312000 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/md0 doesn't contain a valid partition table

Disk /dev/md1: 271 MB, 271319040 bytes
2 heads, 4 sectors/track, 66240 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/md1 doesn't contain a valid partition table

and also

Code: Select all

# df -ah
Filesystem            Size  Used Avail Use% Mounted on
/dev/md0              223G   74G  138G  35% /
proc                     0     0     0   -  /proc
sysfs                    0     0     0   -  /sys
devpts                   0     0     0   -  /dev/pts
/dev/md1              251M   27M  212M  12% /boot
tmpfs                 989M     0  989M   0% /dev/shm
none                     0     0     0   -  /proc/sys/fs/binfmt_misc
tmpfs                 989M     0  989M   0% /usr/local/psa/handlers/before-local
tmpfs                 989M     0  989M   0% /usr/local/psa/handlers/before-queue
tmpfs                 989M     0  989M   0% /usr/local/psa/handlers/before-remote
tmpfs                 989M  1.7M  988M   1% /usr/local/psa/handlers/info
tmpfs                 989M     0  989M   0% /usr/local/psa/handlers/spool

[damon]

Ok, with the help of this article
http://www.linux.com/news/software/appl ... swap-space

I've used the mkswap command to enable swap on the existing swap partition and then turned it on with swapon.

It appears to be working, see below.

The article mentions that the swap partition needs to be added to the fstab file

"To mount the swap space automatically at boot time, you must add an entry to the /etc/fstab file, which contains a list of filesystems and swap spaces that need to be mounted at boot up."

My fstab looks like this

Code: Select all

/dev/md0                /                       ext3    defaults        1 1
/dev/md1                /boot                   ext3    defaults        1 2
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
sysfs                   /sys                    sysfs   defaults        0 0

so it appears that I should add the line

Code: Select all

/dev/sdb2      none    swap    sw      0       0

as the new third line. Would that be correct?

Code: Select all

# swapon -s
Filename                                Type            Size    Used    Priority
/dev/sdb2                               partition       1959920 1972    -1

# cat /proc/meminfo
MemTotal:      2025144 kB
MemFree:         37128 kB
Buffers:         57008 kB
Cached:         695784 kB
SwapCached:         84 kB
Active:        1359500 kB
Inactive:       466952 kB
SwapTotal:     1959920 kB
SwapFree:      1957952 kB
Dirty:             244 kB
Writeback:          80 kB
AnonPages:     1073588 kB
Mapped:          57524 kB
Slab:            99448 kB
SReclaimable:    87272 kB
SUnreclaim:      12176 kB
PageTables:      22908 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
WritebackTmp:        0 kB
CommitLimit:   2972492 kB
Committed_AS:  1642332 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     13984 kB
VmallocChunk: 34359724075 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
HugePages_Surp:      0
Hugepagesize:     2048 kB
DirectMap4k:      3456 kB
DirectMap2M:   2060288 kB