systemd stuck in uninterruptible sleep state

Status changed to 'Confirmed' because the bug affects multiple users.

We just hit a similar issue on Ubuntu 18.04. From the crash dump, I'm able to see this stack:

[12929.914040] kworker/u4:5 D 0 27285 2 0x80000080
[12929.914053] Workqueue: events_unbound fsnotify_mark_destroy_workfn
[12929.914055] Call Trace:
[12929.914075] __schedule+0x291/0x8a0
[12929.914078] schedule+0x2c/0x80
[12929.914081] schedule_timeout+0x1cf/0x350
[12929.914086] ? select_idle_sibling+0x29/0x410
[12929.914089] ? __enqueue_entity+0x5c/0x60
[12929.914091] ? enqueue_entity+0x10e/0x6b0
[12929.914093] wait_for_completion+0xba/0x140
[12929.914094] ? wake_up_q+0x80/0x80
[12929.914099] __synchronize_srcu.part.13+0x85/0xb0
[12929.914101] ? trace_raw_output_rcu_utilization+0x50/0x50
[12929.914103] ? ttwu_do_activate+0x77/0x80
[12929.914105] synchronize_srcu+0x66/0xe0
[12929.914107] ? synchronize_srcu+0x66/0xe0
[12929.914109] fsnotify_mark_destroy_workfn+0x7b/0xe0
[12929.914113] process_one_work+0x1de/0x410
[12929.914115] worker_thread+0x228/0x410
[12929.914118] kthread+0x121/0x140
[12929.914119] ? process_one_work+0x410/0x410
[12929.914121] ? kthread_create_worker_on_cpu+0x70/0x70

Several other systemd processes are also hung. I was able to issue an NMI to generate a crash dump so I can pull out more information if that would be helpful.

4.15.0-43-generic #46~16.04.1-Ubuntu SMP x86_64

I have the same problem but the process that gets stuck is dockerd not systemd.

3,2077,80354225063,-;INFO: task dockerd:2070 blocked for more than 120 seconds.
3,2078,80354230430,-; Tainted: P O 4.15.0-43-generic #46~16.04.1-Ubuntu
3,2079,80354236566,-;"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
6,2080,80354242978,-;dockerd D 0 2070 1 0x00000000
4,2081,80354242981,-;Call Trace:
4,2082,80354242984,-; __schedule+0x3d6/0x8b0
4,2083,80354242986,-; ? xen_smp_send_reschedule+0x10/0x20
4,2084,80354242989,-; schedule+0x36/0x80
4,2085,80354242991,-; schedule_timeout+0x1db/0x370
4,2086,80354242993,-; ? try_to_wake_up+0x59/0x4a0
4,2087,80354242995,-; wait_for_completion+0xb4/0x140
4,2088,80354242996,-; ? wake_up_q+0x70/0x70
4,2089,80354242998,-; flush_work+0x129/0x1e0
4,2090,80354242999,-; ? worker_detach_from_pool+0xb0/0xb0
4,2091,80354243001,-; flush_delayed_work+0x3f/0x50
4,2092,80354243004,-; fsnotify_wait_marks_destroyed+0x15/0x20
4,2093,80354243005,-; fsnotify_destroy_group+0x48/0xd0
4,2094,80354243008,-; inotify_release+0x1e/0x50
4,2095,80354243011,-; __fput+0xea/0x220
4,2096,80354243013,-; ____fput+0xe/0x10
4,2097,80354243014,-; task_work_run+0x8a/0xb0
4,2098,80354243016,-; exit_to_usermode_loop+0xc4/0xd0
4,2099,80354243018,-; do_syscall_64+0xf4/0x130
4,2100,80354243020,-; entry_SYSCALL_64_after_hwframe+0x3d/0xa2

The tainted marker is from zfs module since I have a ZFS partition mounted in the system.
The / partition however is EXT4, and docker is running from /.

cat /proc/1/stack
[<0>] flush_work+0x129/0x1e0
[<0>] flush_delayed_work+0x3f/0x50
[<0>] fsnotify_wait_marks_destroyed+0x15/0x20
[<0>] fsnotify_destroy_group+0x48/0xd0
[<0>] inotify_release+0x1e/0x50
[<0>] __fput+0xea/0x220
[<0>] ____fput+0xe/0x10
[<0>] task_work_run+0x8a/0xb0
[<0>] exit_to_usermode_loop+0xc4/0xd0
[<0>] do_syscall_64+0xf4/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[<0>] 0xffffffffffffffff

6,2111,218820611653,-; task PC stack pid father
6,2112,218820614372,-;systemd D 0 1 0 0x00000004
4,2113,218820616975,-;Call Trace:
4,2114,218820618224,-; __schedule+0x3d6/0x8b0
4,2115,218820619977,-; ? xen_smp_send_reschedule+0x10/0x20
4,2116,218820622101,-; schedule+0x36/0x80
4,2117,218820624470,-; schedule_timeout+0x1db/0x370
4,2118,218820627660,-; ? delete_node+0x1a5/0x1f0
4,2119,218820630763,-; wait_for_completion+0xb4/0x140
4,2120,218820634184,-; ? wake_up_q+0x70/0x70
4,2121,218820638148,-; flush_work+0x129/0x1e0
4,2122,218820642490,-; ? worker_detach_from_pool+0xb0/0xb0
4,2123,218820646706,-; flush_delayed_work+0x3f/0x50
4,2124,218820650474,-; fsnotify_wait_marks_destroyed+0x15/0x20
4,2125,218820654789,-; fsnotify_destroy_group+0x48/0xd0
4,2126,218820658838,-; inotify_release+0x1e/0x50
4,2127,218820662400,-; __fput+0xea/0x220
4,2128,218820665486,-; ____fput+0xe/0x10
4,2129,218820668890,-; task_work_run+0x8a/0xb0
4,2130,218820672633,-; exit_to_usermode_loop+0xc4/0xd0
4,2131,218820676852,-; do_syscall_64+0xf4/0x130
4,2132,218820680778,-; entry_SYSCALL_64_after_hwframe+0x3d/0xa2
4,2133,218820686587,-;RIP: 0033...

I think in my case it was this issue that made systemd go into D state:
https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1802021
I was yet to test the fix in that bug but will do it as soon as it's available on 4.15.+

On 18.04 with kernel 4.15.0-46-generic #49-Ubuntu we hit the hang:

[Apr25 10:10] INFO: task kworker/u256:3:48574 blocked for more than 120 seconds.
[ +0.000106] Not tainted 4.15.0-46-generic #49-Ubuntu
[ +0.000017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ +0.000023] kworker/u256:3 D 0 48574 2 0x80000000
[ +0.000009] Workqueue: events_unbound fsnotify_mark_destroy_workfn
[ +0.000001] Call Trace:
[ +0.000007] __schedule+0x291/0x8a0
[ +0.000002] schedule+0x2c/0x80
[ +0.000002] schedule_timeout+0x1cf/0x350
[ +0.000003] ? select_idle_sibling+0x29/0x410
[ +0.000002] ? __enqueue_entity+0x5c/0x60
[ +0.000002] ? enqueue_entity+0x10e/0x6b0
[ +0.000002] wait_for_completion+0xba/0x140
[ +0.000001] ? wake_up_q+0x80/0x80
[ +0.000004] __synchronize_srcu.part.13+0x85/0xb0
[ +0.000002] ? trace_raw_output_rcu_utilization+0x50/0x50
[ +0.000001] ? ttwu_do_activate+0x77/0x80
[ +0.000002] synchronize_srcu+0x66/0xe0
[ +0.000001] ? synchronize_srcu+0x66/0xe0
[ +0.000002] fsnotify_mark_destroy_workfn+0x7b/0xe0
[ +0.000003] process_one_work+0x1de/0x410
[ +0.000001] worker_thread+0x228/0x410
[ +0.000002] kthread+0x121/0x140
[ +0.000001] ? process_one_work+0x410/0x410
[ +0.000002] ? kthread_create_worker_on_cpu+0x70/0x70
[ +0.000003] ? do_syscall_64+0x73/0x130
[ +0.000002] ? SyS_exit_group+0x14/0x20
[ +0.000002] ret_from_fork+0x35/0x40
[ +0.000008] INFO: task systemd:50609 blocked for more than 120 seconds.
[ +0.000021] Not tainted 4.15.0-46-generic #49-Ubuntu
[ +0.000016] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ +0.000021] systemd D 0 50609 1 0x00000004
[ +0.000002] Call Trace:
[ +0.000002] __schedule+0x291/0x8a0
[ +0.000002] ? sock_def_readable+0x40/0x70
[ +0.000003] ? unix_dgram_sendmsg+0x356/0x6f0
[ +0.000001] schedule+0x2c/0x80
[ +0.000002] schedule_timeout+0x1cf/0x350
[ +0.000003] ? __radix_tree_delete+0x7f/0xa0
[ +0.000001] wait_for_completion+0xba/0x140
[ +0.000001] ? wake_up_q+0x80/0x80
[ +0.000002] flush_work+0x126/0x1e0
[ +0.000001] ? worker_detach_from_pool+0xa0/0xa0
[ +0.000002] flush_delayed_work+0x3f/0x50
[ +0.000002] fsnotify_wait_marks_destroyed+0x15/0x20
[ +0.000001] fsnotify_destroy_group+0x48/0xd0
[ +0.000002] inotify_release+0x1e/0x50
[ +0.000003] __fput+0xea/0x220
[ +0.000001] ____fput+0xe/0x10
[ +0.000002] task_work_run+0x9d/0xc0
[ +0.000002] exit_to_usermode_loop+0xc0/0xd0
[ +0.000001] do_syscall_64+0x115/0x130
[ +0.000002] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[ +0.000002] RIP: 0033:0x7f64c37068d4
[ +0.000001] RSP: 002b:00007ffeaad50378 EFLAGS: 00000246 ORIG_RAX: 0000000000000003
[ +0.000001] RAX: 0000000000000000 RBX: 000055de952eaea0 RCX: 00007f64c37068d4
[ +0.000001] RDX: 000000000000000c RSI: 000055de952eaea0 RDI: 000000000000000c
[ +0.000001] RBP: 00007ffeaad503a0 R08: 000055de952e4598 R09: 0000000000000008
[ +0.000000] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffeaad50398
[ +0.000001] R13: 00007ffeaad504e0 R14: 00007ffeaad50550 R15: 000055de952e2ee0

This bug looks similar/related to:
https://github.com/systemd/systemd/issues/10123
https://www.spinics.net/lists/linux-fsdevel/msg132837.html

I've attached my hung tasks (sysrq w).

please reopen if this is still an issue

For documentation purposes, this issue is likely a duplicate of bug 1802021, fixed in kernel 4.15.0-47.50.