swapper error

[BruceLee]

Had this for the first time. From what I know it could have various roots.
The memory amount have never been a problem. During this time a relatively huge data amount is being transfered via ftp. (backup)
Can anybody tell something about it?
Any feedback is welcome.
Thanks a lot

Apr 17 02:37:18 www kernel: swapper/0: page allocation failure: order:2, mode:0x4020
Apr 17 02:37:18 www kernel: Pid: 0, comm: swapper/0 Tainted: G O 3.2.62-74.art.x86_64 #1
Apr 17 02:37:18 www kernel: Call Trace:
Apr 17 02:37:18 www kernel: <IRQ> [<ffffffff810e1537>] warn_alloc_failed+0x110/0x138
Apr 17 02:37:18 www kernel: [<ffffffff810e1df0>] __alloc_pages_nodemask+0x78b/0x812
Apr 17 02:37:18 www kernel: [<ffffffff814df62b>] ? pax_exit_kernel+0x1b/0x60
Apr 17 02:37:18 www kernel: [<ffffffff811188e6>] kmalloc_large_node+0x62/0x91
Apr 17 02:37:18 www kernel: [<ffffffff8111a5e7>] __kmalloc_node_track_caller+0x32/0x14d
Apr 17 02:37:18 www kernel: [<ffffffff8140dbae>] ? __alloc_skb+0x47/0x161
Apr 17 02:37:18 www kernel: [<ffffffff8140e8b7>] ? skb_copy+0x35/0x9c
Apr 17 02:37:18 www kernel: [<ffffffff8140dbdb>] __alloc_skb+0x74/0x161
Apr 17 02:37:18 www kernel: [<ffffffff8140e8b7>] skb_copy+0x35/0x9c
Apr 17 02:37:18 www kernel: [<ffffffffa01a9ca5>] tg3_start_xmit+0x724/0x91a [tg3]
Apr 17 02:37:18 www kernel: [<ffffffff81417500>] dev_hard_start_xmit+0x3e3/0x584
Apr 17 02:37:18 www kernel: [<ffffffffa01b3130>] ? tg3_ethtool_ops+0x1b0/0x1b0 [tg3]
Apr 17 02:37:18 www kernel: [<ffffffff81434024>] sch_direct_xmit+0x6b/0x149
Apr 17 02:37:18 www kernel: [<ffffffff8143420d>] __qdisc_run+0x10b/0x126
Apr 17 02:37:18 www kernel: [<ffffffff81413c72>] net_tx_action+0x11a/0x191
Apr 17 02:37:18 www kernel: [<ffffffff81051640>] __do_softirq+0xb5/0x186
Apr 17 02:37:19 www kernel: [<ffffffff814e188c>] call_softirq+0x1c/0x30
Apr 17 02:37:19 www kernel: [<ffffffff81003ba7>] do_softirq+0x43/0x7d
Apr 17 02:37:19 www kernel: [<ffffffff81051420>] irq_exit+0x3e/0xa7
Apr 17 02:37:19 www kernel: [<ffffffff814e2038>] do_IRQ+0xa0/0xb7
Apr 17 02:37:19 www kernel: [<ffffffff814d9284>] common_interrupt+0x84/0x84
Apr 17 02:37:19 www kernel: <EOI> [<ffffffff8106ebe0>] ? __hrtimer_start_range_ns+0x2cc/0x2de
Apr 17 02:37:19 www kernel: [<ffffffff8102618c>] ? native_safe_halt+0x6/0x8
Apr 17 02:37:19 www kernel: [<ffffffff81009799>] default_idle+0x4f/0x86
Apr 17 02:37:19 www kernel: [<ffffffff81009b19>] amd_e400_idle+0xf9/0x100
Apr 17 02:37:19 www kernel: [<ffffffff810017c3>] cpu_idle+0xb8/0xee
Apr 17 02:37:19 www kernel: [<ffffffff814be7be>] rest_init+0x72/0x74
Apr 17 02:37:19 www kernel: [<ffffffff85e10d0b>] start_kernel+0x3e0/0x3eb
Apr 17 02:37:19 www kernel: [<ffffffff85e10140>] ? early_idt_handlers+0x140/0x140
Apr 17 02:37:19 www kernel: [<ffffffff85e1026b>] x86_64_start_reservations+0xba/0xbe
Apr 17 02:37:20 www kernel: [<ffffffff85e10136>] ? early_idt_handlers+0x136/0x140
Apr 17 02:37:20 www kernel: [<ffffffff85e10371>] x86_64_start_kernel+0x102/0x111
Apr 17 02:37:20 www kernel: swapper/0: page allocation failure: order:2, mode:0x4020
Apr 17 02:37:20 www kernel: Pid: 0, comm: swapper/0 Tainted: G O 3.2.62-74.art.x86_64 #1
Apr 17 02:37:20 www kernel: Call Trace:
Apr 17 02:37:20 www kernel: <IRQ> [<ffffffff810e1537>] warn_alloc_failed+0x110/0x138
Apr 17 02:37:20 www kernel: [<ffffffff8146e367>] ? tcp_v4_do_rcv+0x1e5/0x437
Apr 17 02:37:20 www kernel: [<ffffffff810e1df0>] __alloc_pages_nodemask+0x78b/0x812
Apr 17 02:37:21 www kernel: [<ffffffff8127d1e7>] ? __list_add+0x1b/0x38
Apr 17 02:37:21 www kernel: [<ffffffff811188e6>] kmalloc_large_node+0x62/0x91
Apr 17 02:37:21 www kernel: [<ffffffff8111a5e7>] __kmalloc_node_track_caller+0x32/0x14d
Apr 17 02:37:21 www kernel: [<ffffffff8140dbae>] ? __alloc_skb+0x47/0x161
Apr 17 02:37:21 www kernel: [<ffffffff8140e8b7>] ? skb_copy+0x35/0x9c
Apr 17 02:37:21 www kernel: [<ffffffff8140dbdb>] __alloc_skb+0x74/0x161
Apr 17 02:37:21 www kernel: [<ffffffff8140e8b7>] skb_copy+0x35/0x9c
Apr 17 02:37:21 www kernel: [<ffffffffa01a9ca5>] tg3_start_xmit+0x724/0x91a [tg3]
Apr 17 02:37:21 www kernel: [<ffffffff81417500>] dev_hard_start_xmit+0x3e3/0x584
Apr 17 02:37:21 www kernel: [<ffffffffa01b3130>] ? tg3_ethtool_ops+0x1b0/0x1b0 [tg3]
Apr 17 02:37:21 www kernel: [<ffffffff81434024>] sch_direct_xmit+0x6b/0x149
Apr 17 02:37:21 www kernel: [<ffffffff8143420d>] __qdisc_run+0x10b/0x126
Apr 17 02:37:21 www kernel: [<ffffffff81413c72>] net_tx_action+0x11a/0x191
Apr 17 02:37:22 www kernel: [<ffffffff81051640>] __do_softirq+0xb5/0x186
Apr 17 02:37:22 www kernel: [<ffffffff814e188c>] call_softirq+0x1c/0x30
Apr 17 02:37:22 www kernel: [<ffffffff81003ba7>] do_softirq+0x43/0x7d
Apr 17 02:37:22 www kernel: [<ffffffff81051420>] irq_exit+0x3e/0xa7
Apr 17 02:37:22 www kernel: [<ffffffff814e2038>] do_IRQ+0xa0/0xb7
Apr 17 02:37:22 www kernel: [<ffffffff814d9284>] common_interrupt+0x84/0x84
Apr 17 02:37:22 www kernel: <EOI> [<ffffffff8106ebe0>] ? __hrtimer_start_range_ns+0x2cc/0x2de
Apr 17 02:37:22 www kernel: [<ffffffff8102618c>] ? native_safe_halt+0x6/0x8
Apr 17 02:37:22 www kernel: [<ffffffff81009799>] default_idle+0x4f/0x86
Apr 17 02:37:22 www kernel: [<ffffffff81009b19>] amd_e400_idle+0xf9/0x100
Apr 17 02:37:22 www kernel: [<ffffffff810017c3>] cpu_idle+0xb8/0xee
Apr 17 02:37:22 www kernel: [<ffffffff814be7be>] rest_init+0x72/0x74
Apr 17 02:37:23 www kernel: [<ffffffff85e10d0b>] start_kernel+0x3e0/0x3eb
Apr 17 02:37:23 www kernel: [<ffffffff85e10140>] ? early_idt_handlers+0x140/0x140
Apr 17 02:37:23 www kernel: [<ffffffff85e1026b>] x86_64_start_reservations+0xba/0xbe
Apr 17 02:37:23 www kernel: [<ffffffff85e10136>] ? early_idt_handlers+0x136/0x140
Apr 17 02:37:23 www kernel: [<ffffffff85e10371>] x86_64_start_kernel+0x102/0x111
Apr 17 02:37:23 www kernel: swapper/0: page allocation failure: order:2, mode:0x4020
Apr 17 02:37:23 www kernel: Pid: 0, comm: swapper/0 Tainted: G O 3.2.62-74.art.x86_64 #1
Apr 17 02:37:23 www kernel: Call Trace:
Apr 17 02:37:23 www kernel: <IRQ> [<ffffffff810e1537>] warn_alloc_failed+0x110/0x138
Apr 17 02:37:24 www kernel: [<ffffffff8146e367>] ? tcp_v4_do_rcv+0x1e5/0x437
Apr 17 02:37:24 www kernel: [<ffffffff810e1df0>] __alloc_pages_nodemask+0x78b/0x812
Apr 17 02:37:24 www kernel: [<ffffffff8146ea81>] ? tcp_v4_rcv+0x4c8/0x79c
Apr 17 02:37:24 www kernel: [<ffffffff811188e6>] kmalloc_large_node+0x62/0x91
Apr 17 02:37:24 www kernel: [<ffffffff8111a5e7>] __kmalloc_node_track_caller+0x32/0x14d
Apr 17 02:37:24 www kernel: [<ffffffff8140dbae>] ? __alloc_skb+0x47/0x161
Apr 17 02:37:24 www kernel: [<ffffffff8140e8b7>] ? skb_copy+0x35/0x9c
Apr 17 02:37:24 www kernel: [<ffffffff8140dbdb>] __alloc_skb+0x74/0x161
Apr 17 02:37:24 www kernel: [<ffffffff8140e8b7>] skb_copy+0x35/0x9c
Apr 17 02:37:24 www kernel: [<ffffffffa01a9ca5>] tg3_start_xmit+0x724/0x91a [tg3]
Apr 17 02:37:24 www kernel: [<ffffffff81417500>] dev_hard_start_xmit+0x3e3/0x584
Apr 17 02:37:24 www kernel: [<ffffffffa01b3130>] ? tg3_ethtool_ops+0x1b0/0x1b0 [tg3]
Apr 17 02:37:24 www kernel: [<ffffffff81434024>] sch_direct_xmit+0x6b/0x149
Apr 17 02:37:24 www kernel: [<ffffffff8143420d>] __qdisc_run+0x10b/0x126
Apr 17 02:37:24 www kernel: [<ffffffff81413c72>] net_tx_action+0x11a/0x191
Apr 17 02:37:25 www kernel: [<ffffffff81051640>] __do_softirq+0xb5/0x186
Apr 17 02:37:25 www kernel: [<ffffffff814e188c>] call_softirq+0x1c/0x30
Apr 17 02:37:25 www kernel: [<ffffffff81003ba7>] do_softirq+0x43/0x7d
Apr 17 02:37:25 www kernel: [<ffffffff81051420>] irq_exit+0x3e/0xa7
Apr 17 02:37:25 www kernel: [<ffffffff814e2038>] do_IRQ+0xa0/0xb7
Apr 17 02:37:25 www kernel: [<ffffffff814d9284>] common_interrupt+0x84/0x84
Apr 17 02:37:25 www kernel: <EOI> [<ffffffff8100872f>] ? native_sched_clock+0x37/0x39
Apr 17 02:37:25 www kernel: [<ffffffff8102618c>] ? native_safe_halt+0x6/0x8
Apr 17 02:37:25 www kernel: [<ffffffff81009799>] default_idle+0x4f/0x86
Apr 17 02:37:25 www kernel: [<ffffffff81009b19>] amd_e400_idle+0xf9/0x100
Apr 17 02:37:25 www kernel: [<ffffffff810017c3>] cpu_idle+0xb8/0xee
Apr 17 02:37:25 www kernel: [<ffffffff814be7be>] rest_init+0x72/0x74
Apr 17 02:37:25 www kernel: [<ffffffff85e10d0b>] start_kernel+0x3e0/0x3eb
Apr 17 02:37:25 www kernel: [<ffffffff85e10140>] ? early_idt_handlers+0x140/0x140
Apr 17 02:37:25 www kernel: [<ffffffff85e1026b>] x86_64_start_reservations+0xba/0xbe
Apr 17 02:37:26 www kernel: [<ffffffff85e10136>] ? early_idt_handlers+0x136/0x140
Apr 17 02:37:26 www kernel: [<ffffffff85e10371>] x86_64_start_kernel+0x102/0x111

[mikeshinn]

Many things can cause this, can you tell us what kernel you are using and distro?

Some kernels and distros are better at handling conditions where memory becomes tight, with some distros and kernels you can tune these parameters to help:

Increase vm.min_free_kbytes value, for example to a higher value than a single allocation request.
Change vm.zone_reclaim_mode to 1 if it's set to zero, so the system can reclaim back memory from cached memory.

Both settings can be set in /etc/sysctl.conf, and loaded using sysctl -p /etc/sysctl.conf.

Please note that allocation failures can still happen no matter what you do if memory gets really tight on any Linux systems with any kernel or distro. For example, when a larger burst of GFP_ATOMIC allocations occur which kswapd may struggle to keep up with. However, these allocations should eventually succeed, so if you app eventually worked then you can ignore these.

[BruceLee]

Thanks for the info. I will play with these settings. tonight the error didn't occure again.
It's a CentOS5 system with 3.2.62-74.art.x86_64