On Ubuntu 22.04, 5.15.0-83-generic #92-Ubuntu - our storage system ran into this bug. mdcheck ran for the scheduled 1st day of the month and then hung 6 hours later.
Oct 1 06:52:13 server1 systemd[1]: Starting MD array scrubbing...
Oct 1 06:52:13 server1 root: mdcheck start checking /dev/md0
Oct 1 06:52:13 server1 kernel: [2129098.393495] md: data-check of RAID array md0
Oct 1 12:57:49 server1 kernel: [2151034.623372] INFO: task dmcrypt_write/2:1783 blocked for more than 241 seconds.
Oct 1 12:57:49 server1 kernel: [2151034.623446] Tainted: G S 5.15.0-83-generic #92-Ubuntu
Oct 1 12:57:49 server1 kernel: [2151034.623498] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 1 12:57:49 server1 kernel: [2151034.623559] task:dmcrypt_write/2 state:D stack: 0 pid: 1783 ppid: 2 flags:0x00004000
Oct 1 12:57:49 server1 kernel: [2151034.623566] Call Trace:
Oct 1 12:57:49 server1 kernel: [2151034.623570]
Oct 1 12:57:49 server1 kernel: [2151034.623574] __schedule+0x24e/0x590
Oct 1 12:57:49 server1 kernel: [2151034.623585] ? __schedule+0x256/0x590
Oct 1 12:57:49 server1 kernel: [2151034.623590] schedule+0x69/0x110
Oct 1 12:57:49 server1 kernel: [2151034.623596] md_write_start.part.0+0x174/0x220
Oct 1 12:57:49 server1 kernel: [2151034.623601] ? wait_woken+0x70/0x70
Oct 1 12:57:49 server1 kernel: [2151034.623610] md_write_start+0x14/0x30
Oct 1 12:57:49 server1 kernel: [2151034.623615] raid5_make_request+0x77/0x540 [raid456]
Oct 1 12:57:49 server1 kernel: [2151034.623633] ? cgroup_rstat_updated+0x11c/0x1e0
Oct 1 12:57:49 server1 kernel: [2151034.623642] ? wait_woken+0x70/0x70
Oct 1 12:57:49 server1 kernel: [2151034.623648] md_handle_request+0x12d/0x1b0
Oct 1 12:57:49 server1 kernel: [2151034.623657] ? submit_bio_checks+0x1a5/0x560
Oct 1 12:57:49 server1 kernel: [2151034.623664] md_submit_bio+0x76/0xc0
Oct 1 12:57:49 server1 kernel: [2151034.623670] __submit_bio+0x1a5/0x220
Oct 1 12:57:49 server1 kernel: [2151034.623675] ? psi_task_switch+0xc6/0x220
Oct 1 12:57:49 server1 kernel: [2151034.623682] __submit_bio_noacct+0x85/0x200
Oct 1 12:57:49 server1 kernel: [2151034.623687] submit_bio_noacct+0x4e/0x120
Oct 1 12:57:49 server1 kernel: [2151034.623691] ? schedule+0x69/0x110
Oct 1 12:57:49 server1 kernel: [2151034.623698] dmcrypt_write+0x104/0x130 [dm_crypt]
Oct 1 12:57:49 server1 kernel: [2151034.623708] ? crypt_ctr+0x600/0x600 [dm_crypt]
Oct 1 12:57:49 server1 kernel: [2151034.623715] kthread+0x12a/0x150
Oct 1 12:57:49 server1 kernel: [2151034.623723] ? set_kthread_struct+0x50/0x50
Oct 1 12:57:49 server1 kernel: [2151034.623730] ret_from_fork+0x22/0x30
Oct 1 12:57:49 server1 kernel: [2151034.623739]
Oct 1 12:57:49 server1 kernel: [2151034.623778] INFO: task mdcheck:2323903 blocked for more than 241 seconds.
Oct 1 12:57:49 server1 kernel: [2151034.623833] Tainted: G S 5.15.0-83-generic #92-Ubuntu
Oct 1 12:57:49 server1 kernel: [2151034.625482] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 1 12:57:49 server1 kernel: [2151034.627098] task:mdcheck state:D stack: 0 pid:2323903 ppid: 1 flags:0x00000002
Oct 1 12:57:49 server1 kernel: [2151034.627104] Call Trace:
Oct 1 12:57:49 server1 kernel: [2151034.627106]
Oct 1 12:57:49 server1 kernel: [2151034.627109] __schedule+0x24e/0x590
Oct 1 12:57:49 server1 kernel: [2151034.627114] ? select_idle_sibling+0x2b/0xa60
Oct 1 12:57:49 server1 kernel: [2151034.627124] schedule+0x69/0x110
Oct 1 12:57:49 server1 kernel: [2151034.627129] schedule_timeout+0x103/0x140
Oct 1 12:57:49 server1 kernel: [2151034.627135] ? ttwu_queue_wakelist+0x131/0x1c0
Oct 1 12:57:49 server1 kernel: [2151034.627142] __wait_for_common+0xae/0x150
Oct 1 12:57:49 server1 kernel: [2151034.627148] ? usleep_range_state+0x90/0x90
Oct 1 12:57:49 server1 kernel: [2151034.627155] wait_for_completion+0x24/0x30
Oct 1 12:57:49 server1 kernel: [2151034.627160] kthread_stop+0x6d/0x170
Oct 1 12:57:49 server1 kernel: [2151034.627168] md_unregister_thread+0x44/0x90
Oct 1 12:57:49 server1 kernel: [2151034.627172] md_reap_sync_thread+0x24/0x230
Oct 1 12:57:49 server1 kernel: [2151034.627177] action_store+0x16f/0x300
Oct 1 12:57:49 server1 kernel: [2151034.627182] md_attr_store+0x95/0xf0
Oct 1 12:57:49 server1 kernel: [2151034.627187] sysfs_kf_write+0x3e/0x50
Oct 1 12:57:49 server1 kernel: [2151034.627194] kernfs_fop_write_iter+0x13b/0x1c0
Oct 1 12:57:49 server1 kernel: [2151034.627199] new_sync_write+0x114/0x1a0
Oct 1 12:57:49 server1 kernel: [2151034.627207] vfs_write+0x1d5/0x270
Oct 1 12:57:49 server1 kernel: [2151034.627212] ksys_write+0x67/0xf0
Oct 1 12:57:49 server1 kernel: [2151034.627219] __x64_sys_write+0x19/0x20
Oct 1 12:57:49 server1 kernel: [2151034.627225] do_syscall_64+0x5c/0xc0
Oct 1 12:57:49 server1 kernel: [2151034.627232] ? do_syscall_64+0x69/0xc0
Oct 1 12:57:49 server1 kernel: [2151034.627258] ? do_syscall_64+0x69/0xc0
Oct 1 12:57:49 server1 kernel: [2151034.627263] entry_SYSCALL_64_after_hwframe+0x61/0xcb
Oct 1 12:57:49 server1 kernel: [2151034.627271] RIP: 0033:0x7fb3bd5e8a37
Oct 1 12:57:49 server1 kernel: [2151034.627276] RSP: 002b:00007ffc02410e18 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
Oct 1 12:57:49 server1 kernel: [2151034.627282] RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 00007fb3bd5e8a37
Oct 1 12:57:49 server1 kernel: [2151034.627285] RDX: 0000000000000005 RSI: 0000564ec7e41690 RDI: 0000000000000001
Oct 1 12:57:49 server1 kernel: [2151034.627288] RBP: 0000564ec7e41690 R08: 00007fb3bd6a5460 R09: 000000007fffffff
Oct 1 12:57:49 server1 kernel: [2151034.627291] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000005
Oct 1 12:57:49 server1 kernel: [2151034.627294] R13: 00007fb3bd6ee780 R14: 00007fb3bd6ea600 R15: 00007fb3bd6e9a00
Oct 1 12:57:49 server1 kernel: [2151034.627298]
Oct 1 12:57:49 server1 kernel: [2151034.627300] INFO: task md0_resync:2323914 blocked for more than 241 seconds.
Oct 1 12:57:49 server1 kernel: [2151034.628914] Tainted: G S 5.15.0-83-generic #92-Ubuntu
Oct 1 12:57:49 server1 kernel: [2151034.630539] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 1 12:57:49 server1 kernel: [2151034.632209] task:md0_resync state:D stack: 0 pid:2323914 ppid: 2 flags:0x00004000
Oct 1 12:57:49 server1 kernel: [2151034.632216] Call Trace:
Oct 1 12:57:49 server1 kernel: [2151034.632218]
Oct 1 12:57:49 server1 kernel: [2151034.632220] __schedule+0x24e/0x590
Oct 1 12:57:49 server1 kernel: [2151034.632226] ? __wake_up_common_lock+0x8a/0xc0
Oct 1 12:57:49 server1 kernel: [2151034.632232] schedule+0x69/0x110
Oct 1 12:57:49 server1 kernel: [2151034.632238] raid5_get_active_stripe+0x30e/0x3c0 [raid456]
Oct 1 12:57:49 server1 kernel: [2151034.632253] ? wait_woken+0x70/0x70
Oct 1 12:57:49 server1 kernel: [2151034.632260] raid5_sync_request+0x38c/0x3b0 [raid456]
Oct 1 12:57:49 server1 kernel: [2151034.632274] ? __bpf_trace_tick_stop+0x20/0x20
Oct 1 12:57:49 server1 kernel: [2151034.632281] md_do_sync.cold+0x42f/0xa17
Oct 1 12:57:49 server1 kernel: [2151034.632287] ? wait_woken+0x70/0x70
Oct 1 12:57:49 server1 kernel: [2151034.632295] md_thread+0xad/0x170
Oct 1 12:57:49 server1 kernel: [2151034.632305] ? md_write_inc+0x60/0x60
Oct 1 12:57:49 server1 kernel: [2151034.632312] kthread+0x12a/0x150
Oct 1 12:57:49 server1 kernel: [2151034.632320] ? set_kthread_struct+0x50/0x50
Oct 1 12:57:49 server1 kernel: [2151034.632327] ret_from_fork+0x22/0x30
Oct 1 12:57:49 server1 kernel: [2151034.632336]
Oct 1 12:57:49 server1 kernel: [2151034.632342] INFO: task kworker/7:1:2326463 blocked for more than 241 seconds.
Oct 1 12:57:49 server1 kernel: [2151034.633989] Tainted: G S 5.15.0-83-generic #92-Ubuntu
Oct 1 12:57:49 server1 kernel: [2151034.635672] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 1 12:57:49 server1 kernel: [2151034.637344] task:kworker/7:1 state:D stack: 0 pid:2326463 ppid: 2 flags:0x00004000
Oct 1 12:57:49 server1 kernel: [2151034.637351] Workqueue: xfs-sync/dm-0 xfs_log_worker [xfs]
Oct 1 12:57:49 server1 kernel: [2151034.637562] Call Trace:
Oct 1 12:57:49 server1 kernel: [2151034.637564]
Oct 1 12:57:49 server1 kernel: [2151034.637566] __schedule+0x24e/0x590
Oct 1 12:57:49 server1 kernel: [2151034.637573] schedule+0x69/0x110
Oct 1 12:57:49 server1 kernel: [2151034.637578] schedule_timeout+0x103/0x140
Oct 1 12:57:49 server1 kernel: [2151034.637585] ? ttwu_queue_wakelist+0x131/0x1c0
Oct 1 12:57:49 server1 kernel: [2151034.637591] __wait_for_common+0xae/0x150
Oct 1 12:57:49 server1 kernel: [2151034.637596] ? usleep_range_state+0x90/0x90
Oct 1 12:57:49 server1 kernel: [2151034.637603] wait_for_completion+0x24/0x30
Oct 1 12:57:49 server1 kernel: [2151034.637609] flush_workqueue+0x133/0x3e0
Oct 1 12:57:49 server1 kernel: [2151034.637615] xlog_cil_push_now.isra.0+0x94/0xa0 [xfs]
Oct 1 12:57:49 server1 kernel: [2151034.637802] xlog_cil_force_seq+0x78/0x2b0 [xfs]
Oct 1 12:57:49 server1 kernel: [2151034.637987] ? psi_task_switch+0xc6/0x220
Oct 1 12:57:49 server1 kernel: [2151034.637994] xfs_log_force+0x9e/0x260 [xfs]
Oct 1 12:57:49 server1 kernel: [2151034.638180] xfs_log_worker+0x39/0x90 [xfs]
Oct 1 12:57:49 server1 kernel: [2151034.638364] process_one_work+0x22b/0x3d0
Oct 1 12:57:49 server1 kernel: [2151034.638370] worker_thread+0x53/0x420
Oct 1 12:57:49 server1 kernel: [2151034.638374] ? process_one_work+0x3d0/0x3d0
Oct 1 12:57:49 server1 kernel: [2151034.638378] kthread+0x12a/0x150
Oct 1 12:57:49 server1 kernel: [2151034.638385] ? set_kthread_struct+0x50/0x50
Oct 1 12:57:49 server1 kernel: [2151034.638392] ret_from_fork+0x22/0x30
Oct 1 12:57:49 server1 kernel: [2151034.638401]
Oct 1 12:57:49 server1 kernel: [2151034.638411] INFO: task kworker/u114:38:2342034 blocked for more than 241 seconds.
Oct 1 12:57:49 server1 kernel: [2151034.640574] Tainted: G S 5.15.0-83-generic #92-Ubuntu
Oct 1 12:57:49 server1 kernel: [2151034.642748] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 1 12:57:49 server1 kernel: [2151034.644960] task:kworker/u114:38 state:D stack: 0 pid:2342034 ppid: 2 flags:0x00004000
Oct 1 12:57:49 server1 kernel: [2151034.644967] Workqueue: xfs-cil/dm-0 xlog_cil_push_work [xfs]
Oct 1 12:57:49 server1 kernel: [2151034.645157] Call Trace:
Oct 1 12:57:49 server1 kernel: [2151034.645159]
Oct 1 12:57:49 server1 kernel: [2151034.645161] __schedule+0x24e/0x590
Oct 1 12:57:49 server1 kernel: [2151034.645168] ? submit_bio+0x4a/0x130
Oct 1 12:57:49 server1 kernel: [2151034.645173] schedule+0x69/0x110
Oct 1 12:57:49 server1 kernel: [2151034.645179] xlog_state_get_iclog_space+0x120/0x370 [xfs]
Oct 1 12:57:49 server1 kernel: [2151034.645367] ? wake_up_q+0x90/0x90
Oct 1 12:57:49 server1 kernel: [2151034.645375] xlog_write+0x148/0x680 [xfs]
Oct 1 12:57:49 server1 kernel: [2151034.645563] xlog_cil_push_work+0x34b/0x470 [xfs]
Oct 1 12:57:49 server1 kernel: [2151034.645750] process_one_work+0x22b/0x3d0
Oct 1 12:57:49 server1 kernel: [2151034.645755] worker_thread+0x53/0x420
Oct 1 12:57:49 server1 kernel: [2151034.645759] ? process_one_work+0x3d0/0x3d0
Oct 1 12:57:49 server1 kernel: [2151034.645763] kthread+0x12a/0x150
Oct 1 12:57:49 server1 kernel: [2151034.645770] ? set_kthread_struct+0x50/0x50
Oct 1 12:57:49 server1 kernel: [2151034.645776] ret_from_fork+0x22/0x30
Oct 1 12:57:49 server1 kernel: [2151034.645785]