I think I'm running into this too.
Raid 6 array with 4 drives.
Linux pkitty 5.13.0-52-generic #59~20.04.1-Ubuntu SMP Thu Jun 16 21:21:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
I have all filesystems unmounted (so there's no other activity that should be occurring on the drives).
Once it hangs, even trying to do
echo idle /sys/block/md0/md/sync_action
hangs and the entire array locks up.
It's not a matter of if it will happen, but when.
Is there any way to backport the fixes? 5.19 is a long way away.
kern.log snippet:
ul 5 19:16:36 pkitty kernel: [ 4471.805561] INFO: task scsi_eh_1:276 blocked for more than 120 seconds.
Jul 5 19:16:36 pkitty kernel: [ 4471.805573] Tainted: G OE 5.13.0-52-generic #59~20.04.1-Ubuntu
Jul 5 19:16:36 pkitty kernel: [ 4471.805577] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 5 19:16:36 pkitty kernel: [ 4471.805580] task:scsi_eh_1 state:D stack: 0 pid: 276 ppid: 2 flags:0x00004000
Jul 5 19:16:36 pkitty kernel: [ 4471.805588] Call Trace:
Jul 5 19:16:36 pkitty kernel: [ 4471.805591]
Jul 5 19:16:36 pkitty kernel: [ 4471.805596] __schedule+0x2ee/0x900
Jul 5 19:16:36 pkitty kernel: [ 4471.805605] schedule+0x4f/0xc0
Jul 5 19:16:36 pkitty kernel: [ 4471.805609] schedule_preempt_disabled+0xe/0x10
Jul 5 19:16:36 pkitty kernel: [ 4471.805614] __mutex_lock.isra.0+0x183/0x4d0
Jul 5 19:16:36 pkitty kernel: [ 4471.805618] ? dequeue_entity+0xdb/0x410
Jul 5 19:16:36 pkitty kernel: [ 4471.805627] ? blk_mq_find_and_get_req+0x4f/0x90
Jul 5 19:16:36 pkitty kernel: [ 4471.805634] __mutex_lock_slowpath+0x13/0x20
Jul 5 19:16:36 pkitty kernel: [ 4471.805639] mutex_lock+0x32/0x40
Jul 5 19:16:36 pkitty kernel: [ 4471.805645] device_reset+0x22/0x50 [usb_storage]
Jul 5 19:16:36 pkitty kernel: [ 4471.805654] scsi_eh_ready_devs+0x580/0xa50
Jul 5 19:16:36 pkitty kernel: [ 4471.805683] kthread+0x12b/0x150
Jul 5 19:16:36 pkitty kernel: [ 4471.805687] ? set_kthread_struct+0x40/0x40
Jul 5 19:16:36 pkitty kernel: [ 4471.805692] ret_from_fork+0x22/0x30
Jul 5 19:16:36 pkitty kernel: [ 4471.805701]
Jul 5 19:16:36 pkitty kernel: [ 4471.805704] INFO: task usb-storage:278 blocked for more than 241 seconds.
Jul 5 19:16:36 pkitty kernel: [ 4471.805708] Tainted: G OE 5.13.0-52-generic #59~20.04.1-Ubuntu
Jul 5 19:16:36 pkitty kernel: [ 4471.805711] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 5 19:16:36 pkitty kernel: [ 4471.805713] task:usb-storage state:D stack: 0 pid: 278 ppid: 2 flags:0x00004000
Jul 5 19:16:36 pkitty kernel: [ 4471.805718] Call Trace:
Jul 5 19:16:36 pkitty kernel: [ 4471.805720]
Jul 5 19:16:36 pkitty kernel: [ 4471.805722] __schedule+0x2ee/0x900
Jul 5 19:16:36 pkitty kernel: [ 4471.805727] ? usleep_range_state+0x90/0x90
Jul 5 19:16:36 pkitty kernel: [ 4471.805733] schedule+0x4f/0xc0
Jul 5 19:16:36 pkitty kernel: [ 4471.805736] schedule_timeout+0x202/0x290
Jul 5 19:16:36 pkitty kernel: [ 4471.805742] ? xhci_urb_enqueue+0x1e4/0x520
Jul 5 19:16:36 pkitty kernel: [ 4471.805746] ? usleep_range_state+0x90/0x90
Jul 5 19:16:36 pkitty kernel: [ 4471.805751] __wait_for_common+0xba/0x160
Jul 5 19:16:36 pkitty kernel: [ 4471.805756] wait_for_completion+0x24/0x30
Jul 5 19:16:36 pkitty kernel: [ 4471.805760] usb_sg_wait+0xea/0x170
Jul 5 19:16:36 pkitty kernel: [ 4471.805767] usb_stor_bulk_transfer_sglist+0x97/0xe0 [usb_storage]
Jul 5 19:16:36 pkitty kernel: [ 4471.805776] usb_stor_bulk_srb+0x3d/0x70 [usb_storage]
Jul 5 19:16:36 pkitty kernel: [ 4471.805783] usb_stor_Bulk_transport+0x177/0x410 [usb_storage]
Jul 5 19:16:36 pkitty kernel: [ 4471.805790] ? schedule_timeout+0x202/0x290
Jul 5 19:16:36 pkitty kernel: [ 4471.805796] usb_stor_invoke_transport+0x3e/0x520 [usb_storage]
Jul 5 19:16:36 pkitty kernel: [ 4471.805804] ? usleep_range_state+0x90/0x90
Jul 5 19:16:36 pkitty kernel: [ 4471.805809] ? __wait_for_common+0xfb/0x160
Jul 5 19:16:36 pkitty kernel: [ 4471.805813] ? __raw_callee_save___native_queued_spin_unlock+0x15/0x23
Jul 5 19:16:36 pkitty kernel: [ 4471.805821] usb_stor_transparent_scsi_command+0xe/0x10 [usb_storage]
Jul 5 19:16:36 pkitty kernel: [ 4471.805828] usb_stor_control_thread+0x19b/0x2a0 [usb_storage]
Jul 5 19:16:36 pkitty kernel: [ 4471.805836] ? storage_probe+0x2b0/0x2b0 [usb_storage]
Jul 5 19:16:36 pkitty kernel: [ 4471.805844] kthread+0x12b/0x150
Jul 5 19:16:36 pkitty kernel: [ 4471.805848] ? set_kthread_struct+0x40/0x40
Jul 5 19:16:36 pkitty kernel: [ 4471.805852] ret_from_fork+0x22/0x30
Jul 5 19:16:36 pkitty kernel: [ 4471.805861]
Jul 5 19:16:36 pkitty kernel: [ 4471.805865] INFO: task md0_raid6:419 blocked for more than 241 seconds.
Jul 5 19:16:36 pkitty kernel: [ 4471.805869] Tainted: G OE 5.13.0-52-generic #59~20.04.1-Ubuntu
Jul 5 19:16:36 pkitty kernel: [ 4471.805872] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 5 19:16:36 pkitty kernel: [ 4471.805874] task:md0_raid6 state:D stack: 0 pid: 419 ppid: 2 flags:0x00004000
Jul 5 19:16:36 pkitty kernel: [ 4471.805878] Call Trace:
Jul 5 19:16:36 pkitty kernel: [ 4471.805880]
Jul 5 19:16:36 pkitty kernel: [ 4471.805882] __schedule+0x2ee/0x900
Jul 5 19:16:36 pkitty kernel: [ 4471.805885] ? blk_flush_plug_list+0xe3/0x110
Jul 5 19:16:36 pkitty kernel: [ 4471.805891] schedule+0x4f/0xc0
Jul 5 19:16:36 pkitty kernel: [ 4471.805895] io_schedule+0x16/0x40
Jul 5 19:16:36 pkitty kernel: [ 4471.805899] blk_mq_get_tag+0x11d/0x270
Jul 5 19:16:36 pkitty kernel: [ 4471.805903] ? __wake_up_pollfree+0x40/0x40
Jul 5 19:16:36 pkitty kernel: [ 4471.805910] __blk_mq_alloc_request+0xec/0x120
Jul 5 19:16:36 pkitty kernel: [ 4471.805919] blk_mq_submit_bio+0x108/0x590
Jul 5 19:16:36 pkitty kernel: [ 4471.805924] submit_bio_noacct+0x446/0x510
Jul 5 19:16:36 pkitty kernel: [ 4471.805930] ops_run_io+0x724/0xcb0 [raid456]
Jul 5 19:16:36 pkitty kernel: [ 4471.805943] handle_stripe+0x59f/0x1360 [raid456]
Jul 5 19:16:36 pkitty kernel: [ 4471.805958] handle_active_stripes.isra.0+0x3c5/0x5a0 [raid456]
Jul 5 19:16:36 pkitty kernel: [ 4471.805970] raid5d+0x35d/0x550 [raid456]
Jul 5 19:16:36 pkitty kernel: [ 4471.805981] md_thread+0xb1/0x170
Jul 5 19:16:36 pkitty kernel: [ 4471.805986] ? __wake_up_pollfree+0x40/0x40
Jul 5 19:16:36 pkitty kernel: [ 4471.805992] ? md_start_sync+0x60/0x60
Jul 5 19:16:36 pkitty kernel: [ 4471.805996] kthread+0x12b/0x150
Jul 5 19:16:36 pkitty kernel: [ 4471.806000] ? set_kthread_struct+0x40/0x40
Jul 5 19:16:36 pkitty kernel: [ 4471.806005] ret_from_fork+0x22/0x30
Jul 5 19:16:36 pkitty kernel: [ 4471.806013]
Jul 5 19:16:36 pkitty kernel: [ 4471.806031] INFO: task pool-udisksd:6829 blocked for more than 241 seconds.
Jul 5 19:16:36 pkitty kernel: [ 4471.806035] Tainted: G OE 5.13.0-52-generic #59~20.04.1-Ubuntu
Jul 5 19:16:36 pkitty kernel: [ 4471.806038] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 5 19:16:36 pkitty kernel: [ 4471.806040] task:pool-udisksd state:D stack: 0 pid: 6829 ppid: 1 flags:0x00000000
Jul 5 19:16:36 pkitty kernel: [ 4471.806045] Call Trace:
Jul 5 19:16:36 pkitty kernel: [ 4471.806047]
Jul 5 19:16:36 pkitty kernel: [ 4471.806049] __schedule+0x2ee/0x900
Jul 5 19:16:36 pkitty kernel: [ 4471.806054] schedule+0x4f/0xc0
Jul 5 19:16:36 pkitty kernel: [ 4471.806057] io_schedule+0x16/0x40
Jul 5 19:16:36 pkitty kernel: [ 4471.806061] blk_mq_get_tag+0x11d/0x270
Jul 5 19:16:36 pkitty kernel: [ 4471.806065] ? __wake_up_pollfree+0x40/0x40
Jul 5 19:16:36 pkitty kernel: [ 4471.806072] __blk_mq_alloc_request+0xec/0x120
Jul 5 19:16:36 pkitty kernel: [ 4471.806079] blk_mq_alloc_request+0x7c/0xc0
Jul 5 19:16:36 pkitty kernel: [ 4471.806085] blk_get_request+0x27/0x60
Jul 5 19:16:36 pkitty kernel: [ 4471.806093] sg_io+0xa0/0x3f0
Jul 5 19:16:36 pkitty kernel: [ 4471.806096] ? _copy_from_user+0x2b/0x60
Jul 5 19:16:36 pkitty kernel: [ 4471.806101] ? get_sg_io_hdr+0xc2/0xf0
Jul 5 19:16:36 pkitty kernel: [ 4471.806105] scsi_cmd_ioctl+0x24c/0x2a0
Jul 5 19:16:36 pkitty kernel: [ 4471.806110] scsi_cmd_blk_ioctl+0x51/0x70
Jul 5 19:16:36 pkitty kernel: [ 4471.806114] sd_ioctl_common+0xbb/0x1f0
Jul 5 19:16:36 pkitty kernel: [ 4471.806119] sd_ioctl+0x1c/0x50
Jul 5 19:16:36 pkitty kernel: [ 4471.806124] blkdev_ioctl+0x11c/0x250
Jul 5 19:16:36 pkitty kernel: [ 4471.806128] ? __fget_light+0xce/0xf0
Jul 5 19:16:36 pkitty kernel: [ 4471.806136] block_ioctl+0x3d/0x50
Jul 5 19:16:36 pkitty kernel: [ 4471.806140] __x64_sys_ioctl+0x91/0xc0
Jul 5 19:16:36 pkitty kernel: [ 4471.806146] do_syscall_64+0x61/0xb0
Jul 5 19:16:36 pkitty kernel: [ 4471.806151] ? sysvec_apic_timer_interrupt+0x4e/0x90
Jul 5 19:16:36 pkitty kernel: [ 4471.806157] ? asm_sysvec_apic_timer_interrupt+0xa/0x20
Jul 5 19:16:36 pkitty kernel: [ 4471.806164] entry_SYSCALL_64_after_hwframe+0x44/0xae
Jul 5 19:16:36 pkitty kernel: [ 4471.806170] RIP: 0033:0x7f52345bc3ab
Jul 5 19:16:36 pkitty kernel: [ 4471.806175] RSP: 002b:00007f523356e8e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Jul 5 19:16:36 pkitty kernel: [ 4471.806180] RAX: ffffffffffffffda RBX: 00007f523356ea80 RCX: 00007f52345bc3ab
Jul 5 19:16:36 pkitty kernel: [ 4471.806183] RDX: 00007f523356e900 RSI: 0000000000002285 RDI: 0000000000000012
Jul 5 19:16:36 pkitty kernel: [ 4471.806186] RBP: 0000000000001388 R08: 00007f523356eaa0 R09: 00007f523356f5e8
Jul 5 19:16:36 pkitty kernel: [ 4471.806189] R10: 00007f523356ea10 R11: 0000000000000246 R12: 00007f523356ec10
Jul 5 19:16:36 pkitty kernel: [ 4471.806191] R13: 0000000000000000 R14: 0000000000000012 R15: 00007f523356eaa0
Jul 5 19:16:36 pkitty kernel: [ 4471.806196]
Jul 5 19:16:36 pkitty kernel: [ 4471.806255] INFO: task md0_resync:5122 blocked for more than 241 seconds.
Jul 5 19:16:36 pkitty kernel: [ 4471.806259] Tainted: G OE 5.13.0-52-generic #59~20.04.1-Ubuntu
Jul 5 19:16:36 pkitty kernel: [ 4471.806262] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 5 19:16:36 pkitty kernel: [ 4471.806272] __schedule+0x2ee/0x900
Jul 5 19:16:36 pkitty kernel: [ 4471.806277] schedule+0x4f/0xc0
Jul 5 19:16:36 pkitty kernel: [ 4471.806281] raid5_get_active_stripe+0x449/0x5e0 [raid456]
Jul 5 19:16:36 pkitty kernel: [ 4471.806292] ? __wake_up_pollfree+0x40/0x40
Jul 5 19:16:36 pkitty kernel: [ 4471.806299] raid5_sync_request+0x387/0x3b0 [raid456]
Jul 5 19:16:36 pkitty kernel: [ 4471.806311] md_do_sync.cold+0x3ed/0x953
Jul 5 19:16:36 pkitty kernel: [ 4471.806317] ? __wake_up_pollfree+0x40/0x40
Jul 5 19:16:36 pkitty kernel: [ 4471.806325] md_thread+0xb1/0x170
Jul 5 19:16:36 pkitty kernel: [ 4471.806330] ? md_start_sync+0x60/0x60
Jul 5 19:16:36 pkitty kernel: [ 4471.806334] kthread+0x12b/0x150
Jul 5 19:16:36 pkitty kernel: [ 4471.806338] ? set_kthread_struct+0x40/0x40
Jul 5 19:16:36 pkitty kernel: [ 4471.806342] ret_from_fork+0x22/0x30
Jul 5 19:16:36 pkitty kernel: [ 4471.806351]
[...]
Jul 5 19:18:37 pkitty kernel: [ 4592.635404] INFO: task scsi_eh_1:276 blocked for more than 241 seconds.
Jul 5 19:18:37 pkitty kernel: [ 4592.635412] Tainted: G OE 5.13.0-52-generic #59~20.04.1-Ubuntu
Jul 5 19:18:37 pkitty kernel: [ 4592.635415] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 5 19:18:37 pkitty kernel: [ 4592.635417] task:scsi_eh_1 state:D stack: 0 pid: 276 ppid: 2 flags:0x00004000
Jul 5 19:18:37 pkitty kernel: [ 4592.635423] Call Trace:
Jul 5 19:18:37 pkitty kernel: [ 4592.635425]
Jul 5 19:18:37 pkitty kernel: [ 4592.635428] __schedule+0x2ee/0x900
Jul 5 19:18:37 pkitty kernel: [ 4592.635435] schedule+0x4f/0xc0
Jul 5 19:18:37 pkitty kernel: [ 4592.635437] schedule_preempt_disabled+0xe/0x10
Jul 5 19:18:37 pkitty kernel: [ 4592.635440] __mutex_lock.isra.0+0x183/0x4d0
Jul 5 19:18:37 pkitty kernel: [ 4592.635457] mutex_lock+0x32/0x40
Jul 5 19:18:37 pkitty kernel: [ 4592.635462] device_reset+0x22/0x50 [usb_storage]
Jul 5 19:18:37 pkitty kernel: [ 4592.635468] scsi_eh_ready_devs+0x580/0xa50
Jul 5 19:18:37 pkitty kernel: [ 4592.635474] ? __pm_runtime_resume+0x60/0x80
Jul 5 19:18:37 pkitty kernel: [ 4592.635477] scsi_error_handler+0x446/0x520
Jul 5 19:18:37 pkitty kernel: [ 4592.635482] ? scsi_eh_get_sense+0x230/0x230
Jul 5 19:18:37 pkitty kernel: [ 4592.635487] kthread+0x12b/0x150
Jul 5 19:18:37 pkitty kernel: [ 4592.635490] ? set_kthread_struct+0x40/0x40
Jul 5 19:18:37 pkitty kernel: [ 4592.635493] ret_from_fork+0x22/0x30
Jul 5 19:18:37 pkitty kernel: [ 4592.635499]