I'm seeing this on Linux box05 2.6.32-02063209-generic #02063209 SMP Wed Feb 24 10:09:53 UTC 2010 x86_64 GNU/Linux
occurs while coping a disk to nfs share on another box
Feb 24 21:11:29 box05 kernel: [ 600.530615] INFO: task flush-0:25:2623 blocked for more than 120 seconds. Feb 24 21:11:29 box05 kernel: [ 600.530618] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 24 21:11:29 box05 kernel: [ 600.530622] flush-0:25 D 0000000000000001 0 2623 2 0x00000000 Feb 24 21:11:29 box05 kernel: [ 600.530628] ffff8800c7bdd110 0000000000000046 0000000000000000 ffff8800c7bddfd8 Feb 24 21:11:29 box05 kernel: [ 600.530634] ffff880129f45b40 0000000000015a80 0000000000015a80 ffff8800c7bddfd8 Feb 24 21:11:29 box05 kernel: [ 600.530640] 0000000000015a80 ffff8800c7bddfd8 0000000000015a80 ffff880129f45f00 Feb 24 21:11:29 box05 kernel: [ 600.530645] Call Trace: Feb 24 21:11:29 box05 kernel: [ 600.530651] [<ffffffff81533c52>] io_schedule+0x42/0x60 Feb 24 21:11:29 box05 kernel: [ 600.530668] [<ffffffffa0c31f39>] nfs_wait_bit_uninterruptible+0x9/0x10 [nfs] Feb 24 21:11:29 box05 kernel: [ 600.530673] [<ffffffff815343d5>] __wait_on_bit+0x55/0x80 Feb 24 21:11:29 box05 kernel: [ 600.530689] [<ffffffffa0c31f30>] ? nfs_wait_bit_uninterruptible+0x0/0x10 [nfs] Feb 24 21:11:29 box05 kernel: [ 600.530706] [<ffffffffa0c31f30>] ? nfs_wait_bit_uninterruptible+0x0/0x10 [nfs] Feb 24 21:11:29 box05 kernel: [ 600.530711] [<ffffffff81534478>] out_of_line_wait_on_bit+0x78/0x90 Feb 24 21:11:29 box05 kernel: [ 600.530716] [<ffffffff8107f760>] ? wake_bit_function+0x0/0x30 Feb 24 21:11:29 box05 kernel: [ 600.530732] [<ffffffffa0c31f28>] nfs_wait_on_request+0x28/0x30 [nfs] Feb 24 21:11:29 box05 kernel: [ 600.530750] [<ffffffffa0c36199>] nfs_wait_on_requests_locked+0x79/0xd0 [nfs] Feb 24 21:11:29 box05 kernel: [ 600.530757] [<ffffffff8111e01b>] ? get_partial_node+0x9b/0xc0 Feb 24 21:11:29 box05 kernel: [ 600.530775] [<ffffffffa0c37253>] nfs_sync_mapping_wait+0xb3/0x140 [nfs] Feb 24 21:11:29 box05 kernel: [ 600.530782] [<ffffffff81437a58>] ? pskb_expand_head+0x58/0x1a0 Feb 24 21:11:29 box05 kernel: [ 600.530800] [<ffffffffa0c37366>] nfs_wb_page_priority+0x86/0xe0 [nfs] Feb 24 21:11:29 box05 kernel: [ 600.530806] [<ffffffff8112744e>] ? page_cgroup_zoneinfo+0x2e/0x40 Feb 24 21:11:29 box05 kernel: [ 600.530823] [<ffffffffa0c373ce>] nfs_wb_page+0xe/0x10 [nfs] Feb 24 21:11:29 box05 kernel: [ 600.530836] [<ffffffffa0c26975>] nfs_release_page+0x35/0x60 [nfs] Feb 24 21:11:29 box05 kernel: [ 600.530842] [<ffffffff810e58eb>] try_to_release_page+0x2b/0x40 Feb 24 21:11:29 box05 kernel: [ 600.530847] [<ffffffff810f27e7>] shrink_page_list+0x4b7/0x550 Feb 24 21:11:29 box05 kernel: [ 600.530852] [<ffffffff810f2abd>] ? isolate_lru_pages+0x23d/0x280 Feb 24 21:11:29 box05 kernel: [ 600.530858] [<ffffffff810f2e43>] shrink_inactive_list+0x2f3/0x700 Feb 24 21:11:29 box05 kernel: [ 600.530864] [<ffffffff81436ba9>] ? __alloc_skb+0x49/0x180 Feb 24 21:11:29 box05 kernel: [ 600.530870] [<ffffffff810e8f23>] ? move_freepages_block+0x73/0x80 Feb 24 21:11:29 box05 kernel: [ 600.530875] [<ffffffff810f32a1>] shrink_list+0x51/0xa0 Feb 24 21:11:29 box05 kernel: [ 600.530880] [<ffffffff810f37fa>] shrink_zone+0x1ea/0x200 Feb 24 21:11:29 box05 kernel: [ 600.530885] [<ffffffff810f4233>] shrink_zones+0x63/0xf0 Feb 24 21:11:29 box05 kernel: [ 600.530891] [<ffffffff810f4330>] do_try_to_free_pages+0x70/0x280 Feb 24 21:11:29 box05 kernel: [ 600.530896] [<ffffffff810f46ac>] try_to_free_pages+0x9c/0xc0 Feb 24 21:11:29 box05 kernel: [ 600.530901] [<ffffffff810f2b00>] ? isolate_pages_global+0x0/0x50 Feb 24 21:11:29 box05 kernel: [ 600.530907] [<ffffffff810ebd1b>] __alloc_pages_slowpath+0x25b/0x540 Feb 24 21:11:29 box05 kernel: [ 600.530913] [<ffffffff810ec146>] __alloc_pages_nodemask+0x146/0x180 Feb 24 21:11:29 box05 kernel: [ 600.530919] [<ffffffff8111a317>] alloc_pages_current+0x87/0xd0 Feb 24 21:11:29 box05 kernel: [ 600.530925] [<ffffffff8111f60c>] allocate_slab+0x11c/0x1b0 Feb 24 21:11:29 box05 kernel: [ 600.530931] [<ffffffff8111f6cb>] new_slab+0x2b/0x190 Feb 24 21:11:29 box05 kernel: [ 600.530935] [<ffffffff811218b1>] __slab_alloc+0x121/0x230 Feb 24 21:11:29 box05 kernel: [ 600.530952] [<ffffffffa0c369a2>] ? nfs_writedata_alloc+0x82/0xb0 [nfs] Feb 24 21:11:29 box05 kernel: [ 600.530970] [<ffffffffa0c369a2>] ? nfs_writedata_alloc+0x82/0xb0 [nfs] Feb 24 21:11:29 box05 kernel: [ 600.530975] [<ffffffff81123186>] __kmalloc+0x1a6/0x1f0 Feb 24 21:11:29 box05 kernel: [ 600.530992] [<ffffffffa0c369a2>] nfs_writedata_alloc+0x82/0xb0 [nfs] Feb 24 21:11:29 box05 kernel: [ 600.531009] [<ffffffffa0c369ed>] nfs_flush_one+0x1d/0xf0 [nfs] Feb 24 21:11:29 box05 kernel: [ 600.531026] [<ffffffffa0c31e38>] nfs_pageio_doio+0x48/0x70 [nfs] Feb 24 21:11:29 box05 kernel: [ 600.531042] [<ffffffffa0c31ea9>] nfs_pageio_complete+0x9/0x10 [nfs] Feb 24 21:11:29 box05 kernel: [ 600.531059] [<ffffffffa0c36724>] nfs_writepages+0xd4/0x120 [nfs] Feb 24 21:11:29 box05 kernel: [ 600.531077] [<ffffffffa0c369d0>] ? nfs_flush_one+0x0/0xf0 [nfs] Feb 24 21:11:29 box05 kernel: [ 600.531082] [<ffffffff8105c536>] ? update_curr+0xb6/0x150 Feb 24 21:11:29 box05 kernel: [ 600.531088] [<ffffffff810ee37c>] do_writepages+0x1c/0x30 Feb 24 21:11:29 box05 kernel: [ 600.531095] [<ffffffff8114d5e7>] writeback_single_inode+0xb7/0x270 Feb 24 21:11:29 box05 kernel: [ 600.531100] [<ffffffff81083989>] ? down_read_trylock+0x9/0x10 Feb 24 21:11:29 box05 kernel: [ 600.531106] [<ffffffff8114dde2>] writeback_inodes_wb+0x162/0x210 Feb 24 21:11:29 box05 kernel: [ 600.531112] [<ffffffff810374c9>] ? default_spin_lock_flags+0x9/0x10 Feb 24 21:11:29 box05 kernel: [ 600.531118] [<ffffffff8114dfb6>] wb_writeback+0x126/0x1e0 Feb 24 21:11:29 box05 kernel: [ 600.531124] [<ffffffff8114e16e>] ? wb_work_complete+0x2e/0x60 Feb 24 21:11:29 box05 kernel: [ 600.531129] [<ffffffff8114e23c>] wb_do_writeback+0x2c/0xc0 Feb 24 21:11:29 box05 kernel: [ 600.531135] [<ffffffff8114e308>] bdi_writeback_task+0x38/0xc0 Feb 24 21:11:29 box05 kernel: [ 600.531140] [<ffffffff810fe870>] ? bdi_start_fn+0x0/0x100 Feb 24 21:11:29 box05 kernel: [ 600.531145] [<ffffffff810fe8f7>] bdi_start_fn+0x87/0x100 Feb 24 21:11:29 box05 kernel: [ 600.531150] [<ffffffff810fe870>] ? bdi_start_fn+0x0/0x100 Feb 24 21:11:29 box05 kernel: [ 600.531154] [<ffffffff8107f59e>] kthread+0x8e/0xa0 Feb 24 21:11:29 box05 kernel: [ 600.531159] [<ffffffff8101306a>] child_rip+0xa/0x20 Feb 24 21:11:29 box05 kernel: [ 600.531164] [<ffffffff8107f510>] ? kthread+0x0/0xa0 Feb 24 21:11:29 box05 kernel: [ 600.531168] [<ffffffff81013060>] ? child_rip+0x0/0x20
I'm seeing this on 02063209- generic #02063209 SMP Wed Feb 24 10:09:53 UTC 2010 x86_64 GNU/Linux
Linux box05 2.6.32-
occurs while coping a disk to nfs share on another box
Feb 24 21:11:29 box05 kernel: [ 600.530615] INFO: task flush-0:25:2623 blocked for more than 120 seconds. kernel/ hung_task_ timeout_ secs" disables this message. c52>] io_schedule+ 0x42/0x60 f39>] nfs_wait_ bit_uninterrupt ible+0x9/ 0x10 [nfs] 3d5>] __wait_ on_bit+ 0x55/0x80 f30>] ? nfs_wait_ bit_uninterrupt ible+0x0/ 0x10 [nfs] f30>] ? nfs_wait_ bit_uninterrupt ible+0x0/ 0x10 [nfs] 478>] out_of_ line_wait_ on_bit+ 0x78/0x90 760>] ? wake_bit_ function+ 0x0/0x30 f28>] nfs_wait_ on_request+ 0x28/0x30 [nfs] 199>] nfs_wait_ on_requests_ locked+ 0x79/0xd0 [nfs] 01b>] ? get_partial_ node+0x9b/ 0xc0 253>] nfs_sync_ mapping_ wait+0xb3/ 0x140 [nfs] a58>] ? pskb_expand_ head+0x58/ 0x1a0 366>] nfs_wb_ page_priority+ 0x86/0xe0 [nfs] 44e>] ? page_cgroup_ zoneinfo+ 0x2e/0x40 3ce>] nfs_wb_ page+0xe/ 0x10 [nfs] 975>] nfs_release_ page+0x35/ 0x60 [nfs] 8eb>] try_to_ release_ page+0x2b/ 0x40 7e7>] shrink_ page_list+ 0x4b7/0x550 abd>] ? isolate_ lru_pages+ 0x23d/0x280 e43>] shrink_ inactive_ list+0x2f3/ 0x700 ba9>] ? __alloc_ skb+0x49/ 0x180 f23>] ? move_freepages_ block+0x73/ 0x80 2a1>] shrink_ list+0x51/ 0xa0 7fa>] shrink_ zone+0x1ea/ 0x200 233>] shrink_ zones+0x63/ 0xf0 330>] do_try_ to_free_ pages+0x70/ 0x280 6ac>] try_to_ free_pages+ 0x9c/0xc0 b00>] ? isolate_ pages_global+ 0x0/0x50 d1b>] __alloc_ pages_slowpath+ 0x25b/0x540 146>] __alloc_ pages_nodemask+ 0x146/0x180 317>] alloc_pages_ current+ 0x87/0xd0 60c>] allocate_ slab+0x11c/ 0x1b0 6cb>] new_slab+0x2b/0x190 8b1>] __slab_ alloc+0x121/ 0x230 9a2>] ? nfs_writedata_ alloc+0x82/ 0xb0 [nfs] 9a2>] ? nfs_writedata_ alloc+0x82/ 0xb0 [nfs] 186>] __kmalloc+ 0x1a6/0x1f0 9a2>] nfs_writedata_ alloc+0x82/ 0xb0 [nfs] 9ed>] nfs_flush_ one+0x1d/ 0xf0 [nfs] e38>] nfs_pageio_ doio+0x48/ 0x70 [nfs] ea9>] nfs_pageio_ complete+ 0x9/0x10 [nfs] 724>] nfs_writepages+ 0xd4/0x120 [nfs] 9d0>] ? nfs_flush_ one+0x0/ 0xf0 [nfs] 536>] ? update_ curr+0xb6/ 0x150 37c>] do_writepages+ 0x1c/0x30 5e7>] writeback_ single_ inode+0xb7/ 0x270 989>] ? down_read_ trylock+ 0x9/0x10 de2>] writeback_ inodes_ wb+0x162/ 0x210 4c9>] ? default_ spin_lock_ flags+0x9/ 0x10 fb6>] wb_writeback+ 0x126/0x1e0 16e>] ? wb_work_ complete+ 0x2e/0x60 23c>] wb_do_writeback +0x2c/0xc0 308>] bdi_writeback_ task+0x38/ 0xc0 870>] ? bdi_start_ fn+0x0/ 0x100 8f7>] bdi_start_ fn+0x87/ 0x100 870>] ? bdi_start_ fn+0x0/ 0x100 59e>] kthread+0x8e/0xa0 06a>] child_rip+0xa/0x20 510>] ? kthread+0x0/0xa0 060>] ? child_rip+0x0/0x20
Feb 24 21:11:29 box05 kernel: [ 600.530618] "echo 0 > /proc/sys/
Feb 24 21:11:29 box05 kernel: [ 600.530622] flush-0:25 D 0000000000000001 0 2623 2 0x00000000
Feb 24 21:11:29 box05 kernel: [ 600.530628] ffff8800c7bdd110 0000000000000046 0000000000000000 ffff8800c7bddfd8
Feb 24 21:11:29 box05 kernel: [ 600.530634] ffff880129f45b40 0000000000015a80 0000000000015a80 ffff8800c7bddfd8
Feb 24 21:11:29 box05 kernel: [ 600.530640] 0000000000015a80 ffff8800c7bddfd8 0000000000015a80 ffff880129f45f00
Feb 24 21:11:29 box05 kernel: [ 600.530645] Call Trace:
Feb 24 21:11:29 box05 kernel: [ 600.530651] [<ffffffff81533
Feb 24 21:11:29 box05 kernel: [ 600.530668] [<ffffffffa0c31
Feb 24 21:11:29 box05 kernel: [ 600.530673] [<ffffffff81534
Feb 24 21:11:29 box05 kernel: [ 600.530689] [<ffffffffa0c31
Feb 24 21:11:29 box05 kernel: [ 600.530706] [<ffffffffa0c31
Feb 24 21:11:29 box05 kernel: [ 600.530711] [<ffffffff81534
Feb 24 21:11:29 box05 kernel: [ 600.530716] [<ffffffff8107f
Feb 24 21:11:29 box05 kernel: [ 600.530732] [<ffffffffa0c31
Feb 24 21:11:29 box05 kernel: [ 600.530750] [<ffffffffa0c36
Feb 24 21:11:29 box05 kernel: [ 600.530757] [<ffffffff8111e
Feb 24 21:11:29 box05 kernel: [ 600.530775] [<ffffffffa0c37
Feb 24 21:11:29 box05 kernel: [ 600.530782] [<ffffffff81437
Feb 24 21:11:29 box05 kernel: [ 600.530800] [<ffffffffa0c37
Feb 24 21:11:29 box05 kernel: [ 600.530806] [<ffffffff81127
Feb 24 21:11:29 box05 kernel: [ 600.530823] [<ffffffffa0c37
Feb 24 21:11:29 box05 kernel: [ 600.530836] [<ffffffffa0c26
Feb 24 21:11:29 box05 kernel: [ 600.530842] [<ffffffff810e5
Feb 24 21:11:29 box05 kernel: [ 600.530847] [<ffffffff810f2
Feb 24 21:11:29 box05 kernel: [ 600.530852] [<ffffffff810f2
Feb 24 21:11:29 box05 kernel: [ 600.530858] [<ffffffff810f2
Feb 24 21:11:29 box05 kernel: [ 600.530864] [<ffffffff81436
Feb 24 21:11:29 box05 kernel: [ 600.530870] [<ffffffff810e8
Feb 24 21:11:29 box05 kernel: [ 600.530875] [<ffffffff810f3
Feb 24 21:11:29 box05 kernel: [ 600.530880] [<ffffffff810f3
Feb 24 21:11:29 box05 kernel: [ 600.530885] [<ffffffff810f4
Feb 24 21:11:29 box05 kernel: [ 600.530891] [<ffffffff810f4
Feb 24 21:11:29 box05 kernel: [ 600.530896] [<ffffffff810f4
Feb 24 21:11:29 box05 kernel: [ 600.530901] [<ffffffff810f2
Feb 24 21:11:29 box05 kernel: [ 600.530907] [<ffffffff810eb
Feb 24 21:11:29 box05 kernel: [ 600.530913] [<ffffffff810ec
Feb 24 21:11:29 box05 kernel: [ 600.530919] [<ffffffff8111a
Feb 24 21:11:29 box05 kernel: [ 600.530925] [<ffffffff8111f
Feb 24 21:11:29 box05 kernel: [ 600.530931] [<ffffffff8111f
Feb 24 21:11:29 box05 kernel: [ 600.530935] [<ffffffff81121
Feb 24 21:11:29 box05 kernel: [ 600.530952] [<ffffffffa0c36
Feb 24 21:11:29 box05 kernel: [ 600.530970] [<ffffffffa0c36
Feb 24 21:11:29 box05 kernel: [ 600.530975] [<ffffffff81123
Feb 24 21:11:29 box05 kernel: [ 600.530992] [<ffffffffa0c36
Feb 24 21:11:29 box05 kernel: [ 600.531009] [<ffffffffa0c36
Feb 24 21:11:29 box05 kernel: [ 600.531026] [<ffffffffa0c31
Feb 24 21:11:29 box05 kernel: [ 600.531042] [<ffffffffa0c31
Feb 24 21:11:29 box05 kernel: [ 600.531059] [<ffffffffa0c36
Feb 24 21:11:29 box05 kernel: [ 600.531077] [<ffffffffa0c36
Feb 24 21:11:29 box05 kernel: [ 600.531082] [<ffffffff8105c
Feb 24 21:11:29 box05 kernel: [ 600.531088] [<ffffffff810ee
Feb 24 21:11:29 box05 kernel: [ 600.531095] [<ffffffff8114d
Feb 24 21:11:29 box05 kernel: [ 600.531100] [<ffffffff81083
Feb 24 21:11:29 box05 kernel: [ 600.531106] [<ffffffff8114d
Feb 24 21:11:29 box05 kernel: [ 600.531112] [<ffffffff81037
Feb 24 21:11:29 box05 kernel: [ 600.531118] [<ffffffff8114d
Feb 24 21:11:29 box05 kernel: [ 600.531124] [<ffffffff8114e
Feb 24 21:11:29 box05 kernel: [ 600.531129] [<ffffffff8114e
Feb 24 21:11:29 box05 kernel: [ 600.531135] [<ffffffff8114e
Feb 24 21:11:29 box05 kernel: [ 600.531140] [<ffffffff810fe
Feb 24 21:11:29 box05 kernel: [ 600.531145] [<ffffffff810fe
Feb 24 21:11:29 box05 kernel: [ 600.531150] [<ffffffff810fe
Feb 24 21:11:29 box05 kernel: [ 600.531154] [<ffffffff8107f
Feb 24 21:11:29 box05 kernel: [ 600.531159] [<ffffffff81013
Feb 24 21:11:29 box05 kernel: [ 600.531164] [<ffffffff8107f
Feb 24 21:11:29 box05 kernel: [ 600.531168] [<ffffffff81013