kernel io hangs during mdcheck/resync

Hello Chad Wagner,

Thank you for reporting this issue. Could you please try installing the latest 20.04 HWE kernel and check whether the problem persists? The version currently in focal-updates is 5.11.0-34.36~20.04.1.

Hi Kleber,
  I installed it later yesterday, but I won't know until the next resync. This has been a problem since at least linux 5.4 kernel that shipped with Ubuntu 20.04. I don't think I had these problems on Ubuntu 18.04 LTS, the same hardware, running the linux-image-generic at that time.

Similar report here on 5.10.0-rc4:
https://www.spinics.net/lists/raid/msg66654.html

I ended up masking the services introduced with 20.04 LTS, and switched back the crontab.

systemctl mask mdcheck_continue.service mdcheck_continue.timer mdcheck_start.service mdcheck_start.timer
cat > /etc/cron.d/mdadm << 'EOF'
#
# cron.d/mdadm -- schedules periodic redundancy checks of MD devices
#
# Copyright © martin f. krafft <email address hidden>
# distributed under the terms of the Artistic Licence 2.0
#

# By default, run at 00:57 on every Sunday, but do nothing unless the day of
# the month is less than or equal to 7. Thus, only run on the first Sunday of
# each month. crontab(5) sucks, unfortunately, in this regard; therefore this
# hack (see #380425).
57 0 * * 0 root if [ -x /usr/share/mdadm/checkarray ] && [ $(date +\%d) -le 7 ]; then /usr/share/mdadm/checkarray --cron --all --idle --quiet; fi
EOF

The pausing and resuming of the integrity check was an annoyance for me anyways.

Here is the proposed patch, Doesn't appear to have been applied. Last report was with 5.11rc5.

https://lore.kernel<email address hidden>/

I think I've seen this issue once per two different systems so I think this is software issue. Does anybody know if patch in comment #6 is going to be included in Ubuntu 20.04 LTS?

Status changed to 'Confirmed' because the bug affects multiple users.

One of the systems was using package linux-generic and in practice

Linux pedabackup 5.4.0-80-generic #90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

$ cat /proc/version_signature
Ubuntu 5.4.0-80.90-generic 5.4.124

The patch hasn't made it into mainline from what I have seen, it looks like it died back in March waiting for feedback from additional kernel developers. From what I have gathered this is a deadlock scenario directly caused by pausing the resync while the system is under heavy write activity.

Donald Buczek provided a reproducer which is a shell script that generates a lot of write activity and pauses/resumes the raid scrubbing. And he also provided a workaround to get the stuck system running without reboot:

echo active > /sys/block/md1/md/array_state

I haven't tried the patch or any of this, I pretty much eliminated the trigger which is mdcheck_start & mdcheck_continue and went back to the 18.04 LTS way of scrubbing arrays (which is basically don't pause/interrupt it once it starts). I ran through a checkarray yesterday to 100%, no problems. Meanwhile since upgrading to 20.04 LTS it has hung almost every single time through 5.4, 5.8, and 5.11 kernels.

Here is Donald Buczek's reproducer script. I setup an Ubuntu 20.04 VM with latest linux-image-generic and was able to reproduce it within maybe 10 or 15 minutes. Exactly the same issue.

Filesystem layout built as follows:

# assemble raid devices
mdadm --create /dev/md0 --level=1 --raid-devices=2 --spare-devices=1 /dev/sda2 /dev/sdb2 /dev/sdc2
mdadm --create /dev/md1 --level=5 --raid-devices=5 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3 /dev/sde3

# create PVs, VGs, LVs
pvcreate /dev/md1
vgcreate vg1 /dev/md1
lvcreate --name root --extents 100%FREE vg1

# create filesystems
mkfs.ext4 /dev/md0
mkfs.ext4 /dev/vg1/root

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1942935

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

Same issue on impish 5.13.13 kernel, running in VBox.

Looks like two patches are landing in next to resolve this:
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20220527&id=8b48ec23cc51a4e7c8dbaef5f34ebe67e1a80934
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20220527&id=1e267742283a4b5a8ca65755c44166be27e9aa0f

I think I'm running into this too.

Raid 6 array with 4 drives.

Linux pkitty 5.13.0-52-generic #59~20.04.1-Ubuntu SMP Thu Jun 16 21:21:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

I have all filesystems unmounted (so there's no other activity that should be occurring on the drives).

Once it hangs, even trying to do

echo idle /sys/block/md0/md/sync_action

hangs and the entire array locks up.

It's not a matter of if it will happen, but when.

Is there any way to backport the fixes? 5.19 is a long way away.

kern.log snippet:

ul 5 19:16:36 pkitty kernel: [ 4471.805561] INFO: task scsi_eh_1:276 blocked for more than 120 seconds.
Jul 5 19:16:36 pkitty kernel: [ 4471.805573] Tainted: G OE 5.13.0-52-generic #59~20.04.1-Ubuntu
Jul 5 19:16:36 pkitty kernel: [ 4471.805577] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 5 19:16:36 pkitty kernel: [ 4471.805580] task:scsi_eh_1 state:D stack: 0 pid: 276 ppid: 2 flags:0x00004000
Jul 5 19:16:36 pkitty kernel: [ 4471.805588] Call Trace:
Jul 5 19:16:36 pkitty kernel: [ 4471.805591] <TASK>
Jul 5 19:16:36 pkitty kernel: [ 4471.805596] __schedule+0x2ee/0x900
Jul 5 19:16:36 pkitty kernel: [ 4471.805605] schedule+0x4f/0xc0
Jul 5 19:16:36 pkitty kernel: [ 4471.805609] schedule_preempt_disabled+0xe/0x10
Jul 5 19:16:36 pkitty kernel: [ 4471.805614] __mutex_lock.isra.0+0x183/0x4d0
Jul 5 19:16:36 pkitty kernel: [ 4471.805618] ? dequeue_entity+0xdb/0x410
Jul 5 19:16:36 pkitty kernel: [ 4471.805627] ? blk_mq_find_and_get_req+0x4f/0x90
Jul 5 19:16:36 pkitty kernel: [ 4471.805634] __mutex_lock_slowpath+0x13/0x20
Jul 5 19:16:36 pkitty kernel: [ 4471.805639] mutex_lock+0x32/0x40
Jul 5 19:16:36 pkitty kernel: [ 4471.805645] device_reset+0x22/0x50 [usb_storage]
Jul 5 19:16:36 pkitty kernel: [ 4471.805654] scsi_eh_ready_devs+0x580/0xa50
Jul 5 19:16:36 pkitty kernel: [ 4471.805683] kthread+0x12b/0x150
Jul 5 19:16:36 pkitty kernel: [ 4471.805687] ? set_kthread_struct+0x40/0x40
Jul 5 19:16:36 pkitty kernel: [ 4471.805692] ret_from_fork+0x22/0x30
Jul 5 19:16:36 pkitty kernel: [ 4471.805701] </TASK>
Jul 5 19:16:36 pkitty kernel: [ 4471.805704] INFO: task usb-storage:278 blocked for more than 241 seconds.
Jul 5 19:16:36 pkitty kernel: [ 4471.805708] Tainted: G OE 5.13.0-52-generic #59~20.04.1-Ubuntu
Jul 5 19:16:36 pkitty kernel: [ 4471.805711] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 5 19:16:36 pkitty kernel: [ 4471.805713] task:usb-storage state:D stack: 0 pid: 278 ppid: 2 flags:0x00004000
Jul 5 19:16:36 pkitty kernel: [ 4471.805718] Call Trace:
Jul 5 19:16:36 pkitty kernel: [ 4471.805720] <TASK>
Jul 5 19:16:36 pkitty kernel: [ 4471.805722] __schedule+0x2ee/0x900
Jul 5 19:16:36 pkitty kernel: [ 4471.805727] ? usleep_range_state+0x90/0x90
Jul 5 19:16:36 pkitty kernel: [ 4471.805733] schedule+0x4f/0xc0
Jul 5 19:16:36 pkitty kernel: [ 4471.805736] schedule_timeout+0x202/0x290
Jul 5 19:16:36 pkitty kernel: [ 4471.805742] ? xhci_urb_enqueue+0x1e4/0x520
Jul 5 19:16:36 pkitty kernel: [ 4471.805746] ? usleep_range_state+0x90/0x90
Jul 5 19:16:36 ...

Tried the patch mentioned in comment #27, as well as the one attached to this bug.

Still hangs on check.

I don’t know much about the mdadm code, but why is it grabbing exclusive locks on a check, which is mostly read operations?

I believe to resolve the deadlock you want to do:
echo active > /sys/block/md1/md/array_state

Not "idle". You should see a hung task for mdcheck in there somewhere as well, and it only occurs when the raid is resyncing (md_resync should be running), at least for me I the workaround in comment 5:
https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-5.11/+bug/1942935/comments/5

Haven't had a problem since, upgraded to 22.04 since then as well with 5.15. I am pretty sure the problem is still there, the problem surfaced in 20.04 when they changed the raid consistency check. The new check pauses the check after 8 hours and triggers the deadlock, I just let it run to completion like it did in 18.04.

I have not checked the patches, but it's possible you have a different problem because I don't see in either of two hung process traces any calls to md_ code.

I'm running checkarray manually, I took off all the start and stop stuff like you did.

echo active > /sys/block/md0/md/array_state doesn't fix.

I must have not gotten all the trace last time. I've attached it here.

Digging further, I think I might be running into this bug:

https://<email address hidden>/T/

Yeah, that's the same issue as this one. The issue is the raid is doing a consistency check (mdcheck) and is transitioned to an "idle" state and hits a deadlock that causes all I/O through the md device to block. The workaround is to change the array state back to active.

I made the changes in #5 almost a year ago and no problems, before that it pretty much hung almost every single month when the scheduled consistency check was triggered ever since upgrading to 20.04.

It won’t let me change the state back to active.

Every time I try nothing happens and array_status is always idle.

The second of the patches mentioned in #27 (with git SHA 1e2677...) has, I believe, been backported to Ubuntu kernels 5.15.0-48 and 5.4.0-126.

We've still hit this with Ubuntu Jammy on 5.15.0-53, so I guess the first commit needs to be backported as well.

Turns out my issue was a faulty drive, and the system would lock up when mdadm hit the bad sectors on resync. The issue seemed like it was lower in the blockdev code causing a deadlock.

I replaced the drive and the problem went away.

Comment #5 (https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-5.11/+bug/1942935/comments/5) has been a stable workaround for me (basically revert back to a continuous resync like 18.04).

My newer machines are using ZFS with raidz2 pools.

I hit this bug with Ubuntu 22.04 (Jammy) on Kernel 5.15.0-56

For testing, I have set up a RAID1 and a RAID5 in a VM. I put the disks for each RAID on a separate controller. Based on the 'mdhang' script (see comment #11), I was able to reproduce the error easily.

I ran 2 'mdhang' scripts at the same time, one for RAID1 and one for RAID5. The RAID5 blocked after a short time. The RAID1 continued to run without problems. So it probably only affects the RAID5.

On my production system I have now activated the workaround (see comment #5).

Status changed to 'Confirmed' because the bug affects multiple users.

I hit his bug as well in Ubuntu 22.04 with kernel 5.15.0-67-generic

We have a single RAID 5 on 3 drives for 28T. I'm switching to the workaround from
comment #5.

Jun 4 12:48:11 server1 kernel: [1622699.548591] INFO: task md0_raid5:406 blocked for more than 120 seconds.
Jun 4 12:48:11 server1 kernel: [1622699.556202] Tainted: G OE 5.15.0-67-generic #74-Ubuntu
Jun 4 12:48:11 server1 kernel: [1622699.564101] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 4 12:48:11 server1 kernel: [1622699.573063] task:md0_raid5 state:D stack: 0 pid: 406 ppid: 2 flags:0x00004000
Jun 4 12:48:11 server1 kernel: [1622699.573077] Call Trace:
Jun 4 12:48:11 server1 kernel: [1622699.573081] <TASK>
Jun 4 12:48:11 server1 kernel: [1622699.573087] __schedule+0x24e/0x590
Jun 4 12:48:11 server1 kernel: [1622699.573103] schedule+0x69/0x110
Jun 4 12:48:11 server1 kernel: [1622699.573115] raid5d+0x3d9/0x5f0 [raid456]
Jun 4 12:48:11 server1 kernel: [1622699.573140] ? wait_woken+0x70/0x70
Jun 4 12:48:11 server1 kernel: [1622699.573151] md_thread+0xad/0x170
Jun 4 12:48:11 server1 kernel: [1622699.573162] ? wait_woken+0x70/0x70
Jun 4 12:48:11 server1 kernel: [1622699.573169] ? md_write_inc+0x60/0x60
Jun 4 12:48:11 server1 kernel: [1622699.573176] kthread+0x12a/0x150
Jun 4 12:48:11 server1 kernel: [1622699.573187] ? set_kthread_struct+0x50/0x50
Jun 4 12:48:11 server1 kernel: [1622699.573197] ret_from_fork+0x22/0x30
Jun 4 12:48:11 server1 kernel: [1622699.573212] </TASK>
Jun 4 12:48:11 server1 kernel: [1622699.573231] INFO: task jbd2/dm-0-8:1375 blocked for more than 120 seconds.
Jun 4 12:48:11 server1 kernel: [1622699.581119] Tainted: G OE 5.15.0-67-generic #74-Ubuntu
Jun 4 12:48:11 server1 kernel: [1622699.589004] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 4 12:48:11 server1 kernel: [1622699.597959] task:jbd2/dm-0-8 state:D stack: 0 pid: 1375 ppid: 2 flags:0x00004000
Jun 4 12:48:11 server1 kernel: [1622699.597968] Call Trace:
Jun 4 12:48:11 server1 kernel: [1622699.597970] <TASK>
Jun 4 12:48:11 server1 kernel: [1622699.597973] __schedule+0x24e/0x590
Jun 4 12:48:11 server1 kernel: [1622699.597984] schedule+0x69/0x110
Jun 4 12:48:11 server1 kernel: [1622699.597992] md_write_start.part.0+0x174/0x220
Jun 4 12:48:11 server1 kernel: [1622699.598002] ? wait_woken+0x70/0x70
Jun 4 12:48:11 server1 kernel: [1622699.598024] md_write_start+0x14/0x30
Jun 4 12:48:11 server1 kernel: [1622699.598032] raid5_make_request+0x77/0x540 [raid456]
Jun 4 12:48:11 server1 kernel: [1622699.598051] ? wait_woken+0x70/0x70
Jun 4 12:48:11 server1 kernel: [1622699.598058] md_handle_request+0x12d/0x1b0
Jun 4 12:48:11 server1 kernel: [1622699.598065] ? __blk_queue_split+0xfe/0x200
Jun 4 12:48:11 server1 kernel: [1622699.598075] md_submit_bio+0x71/0xc0
Jun 4 12:48:11 server1 kernel: [1622699.598082] __submit_bio+0x1a5/0x220
Jun 4 12:48:11 server1 kernel: [1622699.598091] ? mempool_alloc_slab+0x17/0x20
Jun 4 12:48:11 server1 kernel: [1622699.598102] __submit_bio_noacct+0x85/0x200
Jun 4 12:48:11 server1 kernel: [1622699.598...

Seeing this bug on Ubuntu 20.04 and Ubuntu 22.04 as well, with both normal and HWE kernels.

To add some more information, this bug seems to randomly appear during the initial RAID 6 creation process as well, where the array is mounted but completely empty and not accessed - so it's likely to originate within the mdadm resync process itself, unrelated to other system I/O operations. Around 1 in 15 arrays will "freeze" during the initial resync process in my experience, so it is not that uncommon unfortunately.

The symptoms are always the same - at some point during resync, the speeds will radically drop to single MB/s levels and continue degrading over time until "echo active > /sys/block/mdX/md/array_state" is issued. A minute or two after running that, the speeds ramp back up to normal.

This bug seems unrelated to hardware configuration, as I've seen it happen across multiple systems with different CPU vendors, HBA models and with different HDD sizes and vendors. Systems which were previously stable under Ubuntu 18.04 started exhibiting freezes after upgrading to 20.04 as well.

It would also seem disabling mdcheck_start and mdcheck_continue is not necessarily the magical bullet in fixing this, it certainly doesn't seem to help with freezing during the initial resync. I have also seen instances of mdadm scheduled resyncs freezing when triggered using the old cronjob method, with both systemd services and timers masked off.

Dmesg from a freshly installed system where the initial resync "froze" approximately 15 hours after the array was created:
mdadm --create /dev/md1 --level=6 --raid-devices=6 /dev/sd[cdefgh]
mkfs.ext4 /dev/md1
mount -o errors=remount-ro /dev/md1 /srv

[Jul 4 00:18] md/raid:md1: not clean -- starting background reconstruction
[ +0.000062] md/raid:md1: device sdh operational as raid disk 5
[ +0.000002] md/raid:md1: device sdg operational as raid disk 4
[ +0.000001] md/raid:md1: device sdf operational as raid disk 3
[ +0.000001] md/raid:md1: device sde operational as raid disk 2
[ +0.000001] md/raid:md1: device sdd operational as raid disk 1
[ +0.000001] md/raid:md1: device sdc operational as raid disk 0
[ +0.002104] md/raid:md1: raid level 6 active with 6 out of 6 devices, algorithm 2
[ +0.048241] md1: detected capacity change from 0 to 72000290684928
[ +0.000066] md: resync of RAID array md1
[Jul 4 00:32] EXT4-fs (md1): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[Jul 4 02:39] perf: interrupt took too long (2522 > 2500), lowering kernel.perf_event_max_sample_rate to 79250
[Jul 4 04:36] perf: interrupt took too long (3155 > 3152), lowering kernel.perf_event_max_sample_rate to 63250
[Jul 4 15:22] INFO: task md1_raid6:5688 blocked for more than 120 seconds.
[ +0.000059] Not tainted 5.4.0-153-generic #170-Ubuntu
[ +0.000032] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ +0.000045] md1_raid6 D 0 5688 2 0x80004000
[ +0.000004] Call Trace:
[ +0.000011] __schedule+0x2e3/0x740
[ +0.000005] schedule+0x42/0xb0
[ +0.000011] raid5d+0x3e6/0x5f0 [raid456]
[ +0.000005] ? try_to_del_timer_sync+0x54/0x80
[ +0.000005] ? schedule_timeout+0x92/0...

I can't add much in terms of data. But this is a +1. My symptoms were virtually identical to Chad Wagner's
This happened july 1st 2023 on my machine, the day the check started. It got stuck at 98%.
Only cure I could find was to reboot the system.

Unfortunately I have now have seen this bug happen with a data-check of RAID kicked off by system and by cron, so I +1 on Filips's comment #42

In my previous post I said I was switching over to cron triggered raid check, which I did, but it seems I have run into the exact same problem again. I got ten 120 second delays logged in syslog and then the whole system became unreachable.

The data-check started about an hour before the delays were logged.

This shows the first 120 delay that was logged.

Jul 31 21:00:01 server1 kernel: [168797.457900] md: data-check of RAID array md0
...
Jul 31 22:20:46 server1 kernel: [173642.734918] INFO: task kswapd0:190 blocked for more than 120 seconds.
Jul 31 22:20:46 server1 kernel: [173642.742246] Tainted: G O 5.15.0-75-generic #82-Ubuntu
Jul 31 22:20:46 server1 kernel: [173642.750039] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 31 22:20:46 server1 kernel: [173642.758894] task:kswapd0 state:D stack: 0 pid: 190 ppid: 2 flags:0x00004000
Jul 31 22:20:46 server1 kernel: [173642.758906] Call Trace:
Jul 31 22:20:46 server1 kernel: [173642.758910] <TASK>
Jul 31 22:20:46 server1 kernel: [173642.758916] __schedule+0x24e/0x590
Jul 31 22:20:46 server1 kernel: [173642.758935] schedule+0x69/0x110
Jul 31 22:20:46 server1 kernel: [173642.758945] md_write_start.part.0+0x174/0x220
Jul 31 22:20:46 server1 kernel: [173642.758957] ? wait_woken+0x70/0x70
Jul 31 22:20:46 server1 kernel: [173642.758969] md_write_start+0x14/0x30
Jul 31 22:20:46 server1 kernel: [173642.758980] raid5_make_request+0x77/0x540 [raid456]
Jul 31 22:20:46 server1 kernel: [173642.759004] ? wait_woken+0x70/0x70
Jul 31 22:20:46 server1 kernel: [173642.759012] md_handle_request+0x12d/0x1b0
Jul 31 22:20:46 server1 kernel: [173642.759020] ? page_index+0x50/0x50
Jul 31 22:20:46 server1 kernel: [173642.759032] ? __blk_queue_split+0xfe/0x210
Jul 31 22:20:46 server1 kernel: [173642.759043] ? page_index+0x50/0x50
Jul 31 22:20:46 server1 kernel: [173642.759052] md_submit_bio+0x76/0xc0
Jul 31 22:20:46 server1 kernel: [173642.759060] __submit_bio+0x1a5/0x220
Jul 31 22:20:46 server1 kernel: [173642.759072] ? __mod_memcg_lruvec_state+0x63/0xe0
Jul 31 22:20:46 server1 kernel: [173642.759082] __submit_bio_noacct+0x85/0x200
Jul 31 22:20:46 server1 kernel: [173642.759093] submit_bio_noacct+0x4e/0x120
Jul 31 22:20:46 server1 kernel: [173642.759102] ? radix_tree_lookup+0xd/0x20
Jul 31 22:20:46 server1 kernel: [173642.759116] ? bio_associate_blkg_from_css+0x1b2/0x310
Jul 31 22:20:46 server1 kernel: [173642.759124] ? page_index+0x50/0x50
Jul 31 22:20:46 server1 kernel: [173642.759133] ? bio_add_page+0x68/0x90
Jul 31 22:20:46 server1 kernel: [173642.759139] submit_bio+0x4a/0x130
Jul 31 22:20:46 server1 kernel: [173642.759149] __swap_writepage+0x1aa/0x440
Jul 31 22:20:46 server1 kernel: [173642.759160] ? __frontswap_store+0x75/0x120
Jul 31 22:20:46 server1 kernel: [173642.759167] swap_writepage+0x32/0xa0
Jul 31 22:20:46 server1 kernel: [173642.759177] pageout+0x100/0x330
Jul 31 22:20:46 server1 kernel: [173642.759189] shrink_page_list+0xbd9/0xf00
Jul 31 22:20:46 server1 kernel: [173...

Status changed to 'Confirmed' because the bug affects multiple users.

Kernel 5.15.0-75 on Ubuntu 22.04, server1 and server2

Just hit this bug with a different server still running systemd to start the RAID check.

The logs indicate there may be more than one RAID check bug out there:
 - a bug with pausing the raid check
 - a bug with the raid check when it hasn't been paused

In the case with my server2 the raid check was started, then 6 hours later is was interrupted as expected with systemd, but about 20 minutes later the RAID locks up. Pause bug?

Aug 6 18:19:42 server2 systemd[1]: Starting MD array scrubbing...
Aug 6 18:19:42 server2 root: mdcheck start checking /dev/md0
Aug 6 18:19:42 server2 kernel: [2619392.036055] md: data-check of RAID array md0
...
Aug 7 00:19:44 server2 kernel: [2640994.200542] md: md0: data-check interrupted.
... (minutes later the problem happens after the check is supposedly interrupted)

Aug 7 00:23:03 server2: [2641193.425737] INFO: task jbd2/dm-0-8:1904 blocked for more than 120 seconds.
Aug 7 00:23:03 server2 kernel: [2641193.433644] Tainted: G OE 5.15.0-75-generic #82-Ubuntu
Aug 7 00:23:03 server2 kernel: [2641193.441533] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 7 00:23:03 server2 kernel: [2641193.450490] task:jbd2/dm-0-8 state:D stack: 0 pid: 1904 ppid: 2 flags:0x00004000
Aug 7 00:23:03 server2 kernel: [2641193.450503] Call Trace:
Aug 7 00:23:03 server2 kernel: [2641193.450507] <TASK>
Aug 7 00:23:03 server2 kernel: [2641193.450513] __schedule+0x24e/0x590
Aug 7 00:23:03 server2 kernel: [2641193.450531] schedule+0x69/0x110
Aug 7 00:23:03 server2 kernel: [2641193.450541] md_write_start.part.0+0x174/0x220
Aug 7 00:23:03 server2 kernel: [2641193.450553] ? wait_woken+0x70/0x70
Aug 7 00:23:03 server2 kernel: [2641193.450575] md_write_start+0x14/0x30
Aug 7 00:23:03 server2 kernel: [2641193.450585] raid5_make_request+0x77/0x540 [raid456]
Aug 7 00:23:03 server2 kernel: [2641193.450609] ? wait_woken+0x70/0x70
Aug 7 00:23:03 server2 kernel: [2641193.450616] md_handle_request+0x12d/0x1b0
Aug 7 00:23:03 server2 kernel: [2641193.450625] ? __blk_queue_split+0xfe/0x210
Aug 7 00:23:03 server2 kernel: [2641193.450636] md_submit_bio+0x76/0xc0
Aug 7 00:23:03 server2 kernel: [2641193.450643] __submit_bio+0x1a5/0x220
Aug 7 00:23:03 server2 kernel: [2641193.450654] ? mempool_alloc_slab+0x17/0x20
Aug 7 00:23:03 server2 kernel: [2641193.450665] __submit_bio_noacct+0x85/0x200
Aug 7 00:23:03 server2 kernel: [2641193.450674] ? kmem_cache_alloc+0x1ab/0x2f0
Aug 7 00:23:03 server2 kernel: [2641193.450687] submit_bio_noacct+0x4e/0x120
Aug 7 00:23:03 server2 kernel: [2641193.450698] submit_bio+0x4a/0x130
Aug 7 00:23:03 server2 kernel: [2641193.450707] submit_bh_wbc+0x18d/0x1c0
Aug 7 00:23:03 server2 kernel: [2641193.450713] submit_bh+0x13/0x20
Aug 7 00:23:03 server2 kernel: [2641193.450718] jbd2_journal_commit_transaction+0x861/0x17a0
Aug 7 00:23:03 server2 kernel: [2641193.450729] ? __update_idle_core+0x93/0x120
Aug 7 00:23:03 server2 kernel: [2641193.450744] kjournald2+0xa9/0x280
Aug 7 00:23:03 server2 kernel: [2641193.450750] ? wait_woken+0x70/0x70
Aug 7 00:23:03 server2 kernel: [2...

On Ubuntu 22.04, 5.15.0-83-generic #92-Ubuntu - our storage system ran into this bug. mdcheck ran for the scheduled 1st day of the month and then hung 6 hours later.

Oct 1 06:52:13 server1 systemd[1]: Starting MD array scrubbing...
Oct 1 06:52:13 server1 root: mdcheck start checking /dev/md0
Oct 1 06:52:13 server1 kernel: [2129098.393495] md: data-check of RAID array md0

Oct 1 12:57:49 server1 kernel: [2151034.623372] INFO: task dmcrypt_write/2:1783 blocked for more than 241 seconds.
Oct 1 12:57:49 server1 kernel: [2151034.623446] Tainted: G S 5.15.0-83-generic #92-Ubuntu
Oct 1 12:57:49 server1 kernel: [2151034.623498] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 1 12:57:49 server1 kernel: [2151034.623559] task:dmcrypt_write/2 state:D stack: 0 pid: 1783 ppid: 2 flags:0x00004000
Oct 1 12:57:49 server1 kernel: [2151034.623566] Call Trace:
Oct 1 12:57:49 server1 kernel: [2151034.623570] <TASK>
Oct 1 12:57:49 server1 kernel: [2151034.623574] __schedule+0x24e/0x590
Oct 1 12:57:49 server1 kernel: [2151034.623585] ? __schedule+0x256/0x590
Oct 1 12:57:49 server1 kernel: [2151034.623590] schedule+0x69/0x110
Oct 1 12:57:49 server1 kernel: [2151034.623596] md_write_start.part.0+0x174/0x220
Oct 1 12:57:49 server1 kernel: [2151034.623601] ? wait_woken+0x70/0x70
Oct 1 12:57:49 server1 kernel: [2151034.623610] md_write_start+0x14/0x30
Oct 1 12:57:49 server1 kernel: [2151034.623615] raid5_make_request+0x77/0x540 [raid456]
Oct 1 12:57:49 server1 kernel: [2151034.623633] ? cgroup_rstat_updated+0x11c/0x1e0
Oct 1 12:57:49 server1 kernel: [2151034.623642] ? wait_woken+0x70/0x70
Oct 1 12:57:49 server1 kernel: [2151034.623648] md_handle_request+0x12d/0x1b0
Oct 1 12:57:49 server1 kernel: [2151034.623657] ? submit_bio_checks+0x1a5/0x560
Oct 1 12:57:49 server1 kernel: [2151034.623664] md_submit_bio+0x76/0xc0
Oct 1 12:57:49 server1 kernel: [2151034.623670] __submit_bio+0x1a5/0x220
Oct 1 12:57:49 server1 kernel: [2151034.623675] ? psi_task_switch+0xc6/0x220
Oct 1 12:57:49 server1 kernel: [2151034.623682] __submit_bio_noacct+0x85/0x200
Oct 1 12:57:49 server1 kernel: [2151034.623687] submit_bio_noacct+0x4e/0x120
Oct 1 12:57:49 server1 kernel: [2151034.623691] ? schedule+0x69/0x110
Oct 1 12:57:49 server1 kernel: [2151034.623698] dmcrypt_write+0x104/0x130 [dm_crypt]
Oct 1 12:57:49 server1 kernel: [2151034.623708] ? crypt_ctr+0x600/0x600 [dm_crypt]
Oct 1 12:57:49 server1 kernel: [2151034.623715] kthread+0x12a/0x150
Oct 1 12:57:49 server1 kernel: [2151034.623723] ? set_kthread_struct+0x50/0x50
Oct 1 12:57:49 server1 kernel: [2151034.623730] ret_from_fork+0x22/0x30
Oct 1 12:57:49 server1 kernel: [2151034.623739] </TASK>
Oct 1 12:57:49 server1 kernel: [2151034.623778] INFO: task mdcheck:2323903 blocked for more than 241 seconds.
Oct 1 12:57:49 server1 kernel: [2151034.623833] Tainted: G S 5.15.0-83-generic #92-Ubuntu
Oct 1 12:57:49 server1 kernel: [2151034.625482] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 1 12:57:49 server1 kernel: [2151034.627098] task:mdcheck state:D stack: 0 pid:232390...

Noticed this issue on Ubuntu 20.04 with a md raid device. System exhibited the same behavior as other users have noted: high CPU usage and terminal locking up until the system is rebooted.

[14715252.569157] INFO: task md1_raid4:1763945 blocked for more than 120 seconds.
[14715252.570228] Not tainted 5.4.0-146-generic #163-Ubuntu
[14715252.571277] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[14715252.572347] md1_raid4 D 0 1763945 2 0x80004000
[14715252.572357] Call Trace:
[14715252.572360] __schedule+0x2e3/0x740
[14715252.572363] schedule+0x42/0xb0
[14715252.572369] raid5d+0x3e6/0x5f0 [raid456]
[14715252.572376] ? schedule_timeout+0x10e/0x160
[14715252.572381] ? __wake_up_pollfree+0x40/0x40
[14715252.572384] md_thread+0x97/0x160
[14715252.572392] ? __wake_up_pollfree+0x40/0x40
[14715252.572394] kthread+0x104/0x140
[14715252.572399] ? md_start_sync+0x60/0x60
[14715252.572403] ? kthread_park+0x90/0x90
[14715252.572405] ret_from_fork+0x35/0x40
[14715252.572430] INFO: task kworker/u64:1:3189415 blocked for more than 120 seconds.