BUG: soft lockup - CPU#0 stuck for 10s!

General Discussion of atomic repo and development projects.

Ask for help here with anything else not covered by other forums.
premierhosting
Forum Regular
Forum Regular
Posts: 257
Joined: Wed Aug 04, 2010 2:52 pm

BUG: soft lockup - CPU#0 stuck for 10s!

Unread post by premierhosting »

I've been having these kernel soft lockups past couple days. I've been unable to change kernels due to the Xen issue. Just figured I'd post it an perhaps someone can explain what's going on, and it might help someone. I'm going to just move my services to a real live server and have ASL kernel.

uname -a
Linux xxxxxxxx.com 2.6.18-164.11.1.el5 #1 SMP Wed Jan 20 07:32:21 EST 2010 x86_64 x86_64 x86_64 GNU/Linux

Code: Select all

Jan 11 12:30:25 cloud1 kernel: BUG: soft lockup - CPU#0 stuck for 10s! [swapper:0]
Jan 11 12:30:25 cloud1 xinetd[13711]: EXIT: smtp status=0 pid=11135 duration=136(sec)
Jan 11 12:30:34 cloud1 kernel: CPU 0:
Jan 11 12:30:35 cloud1 xinetd[13711]: EXIT: smtp status=0 pid=11189 duration=44(sec)
Jan 11 12:30:35 cloud1 kernel: Modules linked in: dm_crypt ecb cbc aes_x86_64 testmgr_cipher testmgr aead crypto_blkcipher crypto_algapi aes_generic cifs fuse tun ipt_LOG ipt_ecn ipt_ECN xt_string xt_connlimit xt_mark xt_tcpmss xt_DSCP xt_MARK xt_multiport xt_pkttype xt_physdev bridge xt_mac xt_limit xt_length xt_conntrack ipt_REDIRECT ipt_recent ipt_owner ipv6 xfrm_nalgo crypto_api iptable_nat ip_nat iptable_mangle ipt_REJECT xt_tcpudp xt_state ip_conntrack nfnetlink iptable_filter ip_tables x_tables dm_mirror dm_multipath scsi_dh video hwmon backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac lp floppy ide_cd joydev i2c_piix4 8139too i2c_core parport_pc parport 8139cp cdrom serio_raw xen_platform_pci mii pcspkr xfs raid456 xor multipath ahci linear sata_nv dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache mptspi mptscsih scsi_transport_spi mptbase sata_sis pata_sis raid1 ext3 jbd uhci_hcd ohci_hcd ehci_hcd ata_piix libata arcmsr 3w_9xxx 3w_xxxx sd_mod scsi_mod
Jan 11 12:30:35 cloud1 xinetd[13711]: EXIT: smtp status=0 pid=11195 duration=36(sec)
Jan 11 12:30:35 cloud1 kernel: Pid: 0, comm: swapper Not tainted 2.6.18-164.11.1.el5 #1
Jan 11 12:30:35 cloud1 xinetd[13711]: EXIT: smtp status=0 pid=11206 duration=34(sec)
Jan 11 12:30:36 cloud1 kernel: RIP: 0010:[<ffffffff80064b50>]  [<ffffffff80064b50>] _spin_unlock_irqrestore+0x8/0x9
Jan 11 12:30:36 cloud1 xinetd[13711]: START: smtp pid=11256 from=178.187.72.234
Jan 11 12:30:36 cloud1 kernel: RSP: 0018:ffffffff8043dd60  EFLAGS: 00000246
Jan 11 12:30:38 cloud1 kernel: RAX: 0000000000000001 RBX: ffffffff804e3898 RCX: 0000000000000000
Jan 11 12:30:38 cloud1 xinetd[13711]: EXIT: smtp status=0 pid=11213 duration=36(sec)
Jan 11 12:30:38 cloud1 kernel: RDX: 00000000000001f0 RSI: 0000000000000246 RDI: ffffffff803e8600
Jan 11 12:30:40 cloud1 kernel: RBP: ffffffff8043dce0 R08: ffff810001000058 R09: 0000000000000e00
Jan 11 12:30:41 cloud1 kernel: R10: ffff81007f39a000 R11: ffffffff8005a3e7 R12: ffffffff8005dc8e
Jan 11 12:30:41 cloud1 kernel: R13: 0000000000000000 R14: ffffffff80077d08 R15: ffffffff8043dce0
Jan 11 12:30:41 cloud1 kernel: FS:  00002b18bcd97f70(0000) GS:ffffffff803c1000(0000) knlGS:00000000f7cf2ad0
Jan 11 12:30:41 cloud1 kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Jan 11 12:30:41 cloud1 kernel: CR2: 0000000001d27a98 CR3: 000000005fc15000 CR4: 00000000000006e0
Jan 11 12:30:42 cloud1 xinetd[13711]: EXIT: smtp status=0 pid=11197 duration=43(sec)
Jan 11 12:30:42 cloud1 kernel: 
Jan 11 12:30:42 cloud1 xinetd[13711]: START: smtp pid=11262 from=66.210.221.100
Jan 11 12:30:42 cloud1 kernel: Call Trace:
Jan 11 12:30:42 cloud1 kernel:  <IRQ>  [<ffffffff8000d7e7>] ide_intr+0x1d1/0x1df
Jan 11 12:30:42 cloud1 kernel:  [<ffffffff80010b81>] handle_IRQ_event+0x51/0xa6
Jan 11 12:30:42 cloud1 kernel:  [<ffffffff800ba2c5>] __do_IRQ+0xa4/0x103
Jan 11 12:30:42 cloud1 kernel:  [<ffffffff8006c9bf>] do_IRQ+0xe7/0xf5
Jan 11 12:30:42 cloud1 kernel:  [<ffffffff8005d615>] ret_from_intr+0x0/0xa
Jan 11 12:30:42 cloud1 kernel:  [<ffffffff8024418d>] rt_check_expire+0xdc/0x21d
Jan 11 12:30:42 cloud1 kernel:  [<ffffffff802440b1>] rt_check_expire+0x0/0x21d
Jan 11 12:30:42 cloud1 kernel:  [<ffffffff800972b4>] run_timer_softirq+0x133/0x1af
Jan 11 12:30:42 cloud1 kernel:  [<ffffffff8001231e>] __do_softirq+0x89/0x133
Jan 11 12:30:42 cloud1 kernel:  [<ffffffff8005e2fc>] call_softirq+0x1c/0x28
Jan 11 12:30:42 cloud1 kernel:  [<ffffffff8006cb3c>] do_softirq+0x2c/0x85
Jan 11 12:30:42 cloud1 kernel:  [<ffffffff8006c9c4>] do_IRQ+0xec/0xf5
Jan 11 12:30:42 cloud1 kernel:  [<ffffffff8006b2f4>] default_idle+0x0/0x50
Jan 11 12:30:42 cloud1 kernel:  [<ffffffff8005d615>] ret_from_intr+0x0/0xa
Jan 11 12:30:43 cloud1 kernel:  <EOI>  [<ffffffff8006b31d>] default_idle+0x29/0x50
Jan 11 12:30:43 cloud1 kernel:  [<ffffffff8004947b>] cpu_idle+0x95/0xb8
Jan 11 12:30:43 cloud1 kernel:  [<ffffffff803fd7fd>] start_kernel+0x220/0x225
Jan 11 12:30:43 cloud1 kernel:  [<ffffffff803fd22f>] _sinittext+0x22f/0x236
Jan 11 12:30:43 cloud1 kernel: 
Jan 11 12:31:05 cloud1 xinetd[13711]: START: smtp pid=11266 from=64.78.150.46
Jan 11 12:31:22 cloud1 kernel: BUG: soft lockup - CPU#0 stuck for 10s! [pdflush:23797]
Jan 11 12:31:22 cloud1 kernel: CPU 0:
Jan 11 12:31:22 cloud1 kernel: Modules linked in: dm_crypt ecb cbc aes_x86_64 testmgr_cipher testmgr aead crypto_blkcipher crypto_algapi aes_generic cifs fuse tun ipt_LOG ipt_ecn ipt_ECN xt_string xt_connlimit xt_mark xt_tcpmss xt_DSCP xt_MARK xt_multiport xt_pkttype xt_physdev bridge xt_mac xt_limit xt_length xt_conntrack ipt_REDIRECT ipt_recent ipt_owner ipv6 xfrm_nalgo crypto_api iptable_nat ip_nat iptable_mangle ipt_REJECT xt_tcpudp xt_state ip_conntrack nfnetlink iptable_filter ip_tables x_tables dm_mirror dm_multipath scsi_dh video hwmon backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac lp floppy ide_cd joydev i2c_piix4 8139too i2c_core parport_pc parport 8139cp cdrom serio_raw xen_platform_pci mii pcspkr xfs raid456 xor multipath ahci linear sata_nv dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache mptspi mptscsih scsi_transport_spi mptbase sata_sis pata_sis raid1 ext3 jbd uhci_hcd ohci_hcd ehci_hcd ata_piix libata arcmsr 3w_9xxx 3w_xxxx sd_mod scsi_mod
Jan 11 12:31:22 cloud1 kernel: Pid: 23797, comm: pdflush Not tainted 2.6.18-164.11.1.el5 #1
Jan 11 12:31:22 cloud1 kernel: RIP: 0010:[<ffffffff80010b72>]  [<ffffffff80010b72>] handle_IRQ_event+0x42/0xa6
Jan 11 12:31:22 cloud1 kernel: RSP: 0000:ffffffff8043de48  EFLAGS: 00000246
Jan 11 12:31:22 cloud1 kernel: RAX: 0000000002010000 RBX: ffff81007e41e740 RCX: 000000000000000e
Jan 11 12:31:22 cloud1 kernel: RDX: ffff81007e41e740 RSI: ffffffff8043deb8 RDI: 000000000000000e
Jan 11 12:31:22 cloud1 kernel: RBP: ffffffff8043ddc0 R08: ffffffff8005e2fc R09: ffff810081f42000
Jan 11 12:31:22 cloud1 kernel: R10: ffffffff8043df98 R11: ffffffff8005a3e7 R12: ffffffff8005dc8e
Jan 11 12:31:22 cloud1 kernel: R13: ffffffff803da180 R14: ffffffff80077d08 R15: ffffffff8043ddc0
Jan 11 12:31:22 cloud1 kernel: FS:  00002ac8589fe240(0000) GS:ffffffff803c1000(0000) knlGS:00000000f7cf2ad0
Jan 11 12:31:26 cloud1 kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Jan 11 12:31:26 cloud1 xinetd[13711]: EXIT: smtp status=0 pid=11256 duration=50(sec)
Jan 11 12:31:26 cloud1 kernel: CR2: 00002ae22a3b9ee0 CR3: 000000006c679000 CR4: 00000000000006e0
Jan 11 12:31:26 cloud1 xinetd[13711]: EXIT: smtp status=0 pid=11262 duration=44(sec)
Jan 11 12:31:26 cloud1 kernel: 
Jan 11 12:31:26 cloud1 xinetd[13711]: START: smtp pid=11271 from=65.98.103.130
Jan 11 12:31:26 cloud1 kernel: Call Trace:
Jan 11 12:31:26 cloud1 xinetd[13711]: START: smtp pid=11272 from=189.13.85.142
Jan 11 12:31:26 cloud1 kernel:  <IRQ>  [<ffffffff800ba2c5>] __do_IRQ+0xa4/0x103
Jan 11 12:31:27 cloud1 kernel:  [<ffffffff8006c9bf>] do_IRQ+0xe7/0xf5
Jan 11 12:31:27 cloud1 kernel:  [<ffffffff8005d615>] ret_from_intr+0x0/0xa
Jan 11 12:31:27 cloud1 kernel:  [<ffffffff8005a3e7>] ide_outsw+0x0/0x9
Jan 11 12:31:27 cloud1 kernel:  [<ffffffff8005e2fc>] call_softirq+0x1c/0x28
Jan 11 12:31:27 cloud1 kernel:  [<ffffffff800122e6>] __do_softirq+0x51/0x133
Jan 11 12:31:27 cloud1 kernel:  [<ffffffff8005e2fc>] call_softirq+0x1c/0x28
Jan 11 12:31:27 cloud1 kernel:  [<ffffffff8006cb3c>] do_softirq+0x2c/0x85
Jan 11 12:31:27 cloud1 kernel:  [<ffffffff8005dc8e>] apic_timer_interrupt+0x66/0x6c
Jan 11 12:31:27 cloud1 kernel:  <EOI>  [<ffffffff8005a3e7>] ide_outsw+0x0/0x9
Jan 11 12:31:27 cloud1 kernel:  [<ffffffff8000c0b2>] __make_request+0x3f0/0x401
Jan 11 12:31:27 cloud1 kernel:  [<ffffffff8001c028>] generic_make_request+0x211/0x228
Jan 11 12:31:28 cloud1 kernel:  [<ffffffff882b5cb7>] :xfs:xfs_trans_read_buf+0x47/0x2af
Jan 11 12:31:28 cloud1 kernel:  [<ffffffff80033437>] submit_bio+0xe4/0xeb
Jan 11 12:31:28 cloud1 kernel:  [<ffffffff882bf357>] :xfs:_xfs_buf_ioapply+0x1ee/0x219
Jan 11 12:31:29 cloud1 kernel:  [<ffffffff882bf3bb>] :xfs:xfs_buf_iorequest+0x39/0x64
Jan 11 12:31:29 cloud1 kernel:  [<ffffffff882c334e>] :xfs:xfs_bdstrat_cb+0x36/0x3a
Jan 11 12:31:29 cloud1 kernel:  [<ffffffff882bf450>] :xfs:xfs_buf_iostart+0x6a/0x8e
Jan 11 12:31:29 cloud1 kernel:  [<ffffffff882a5aeb>] :xfs:xfs_iflush+0x22b/0x2a4
Jan 11 12:31:29 cloud1 kernel:  [<ffffffff882bb72f>] :xfs:xfs_inode_flush+0xbf/0xe1
Jan 11 12:31:29 cloud1 kernel:  [<ffffffff882c5c06>] :xfs:xfs_fs_write_inode+0x36/0x4d
Jan 11 12:31:29 cloud1 kernel:  [<ffffffff8002fcfd>] __writeback_single_inode+0x1e9/0x328
Jan 11 12:31:29 cloud1 kernel:  [<ffffffff882b5279>] :xfs:xfs_trans_first_ail+0x12/0x1d
Jan 11 12:31:29 cloud1 kernel:  [<ffffffff80020d0d>] sync_sb_inodes+0x1b5/0x26f
Jan 11 12:31:29 cloud1 kernel:  [<ffffffff8009fe9f>] keventd_create_kthread+0x0/0xc4
Jan 11 12:31:29 cloud1 kernel:  [<ffffffff8005121a>] writeback_inodes+0x82/0xd8
Jan 11 12:31:29 cloud1 kernel:  [<ffffffff800c8b9c>] wb_kupdate+0xd4/0x14e
Jan 11 12:31:29 cloud1 kernel:  [<ffffffff8005685a>] pdflush+0x0/0x1fb
Jan 11 12:31:30 cloud1 kernel:  [<ffffffff800569ab>] pdflush+0x151/0x1fb
Jan 11 12:31:30 cloud1 kernel:  [<ffffffff800c8ac8>] wb_kupdate+0x0/0x14e
Jan 11 12:31:30 cloud1 kernel:  [<ffffffff80032950>] kthread+0xfe/0x132
Jan 11 12:31:30 cloud1 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Jan 11 12:31:30 cloud1 kernel:  [<ffffffff8009fe9f>] keventd_create_kthread+0x0/0xc4
Jan 11 12:31:30 cloud1 kernel:  [<ffffffff80032852>] kthread+0x0/0x132
Jan 11 12:31:30 cloud1 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Jan 11 12:31:30 cloud1 kernel: 
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8355
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Re: BUG: soft lockup - CPU#0 stuck for 10s!

Unread post by scott »

If you get those in different places all the time (that looked like the netfilter system) then it could be indicative of bad ram.
premierhosting
Forum Regular
Forum Regular
Posts: 257
Joined: Wed Aug 04, 2010 2:52 pm

Re: BUG: soft lockup - CPU#0 stuck for 10s!

Unread post by premierhosting »

Hi Scott,

Yep, it hits lots of different processes. Would it be bad ram on the host OS running Xen?
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8355
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Re: BUG: soft lockup - CPU#0 stuck for 10s!

Unread post by scott »

That sounds like bad ram to me, and yes it would be the ram on the physical system. Sadly magic ram doesnt exist yet. Virtual systems have to play by the same laws of thermodynamics that the rest of us do.
Post Reply