Btrfs blocked/frozen io when running on top of luks encrypted partition

Bug #1922022 reported by Jesper Utoft
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux-hwe-5.8 (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

Issue description
Issue is present on any ubuntu-20.04 kernel, the issue was present when installing using the first iso & it's present now with a fully updated system running on the hwe kernel. But also on the 5.4.0 kernel.

kcryptd freeze waiting for io for a dm-crypt (luks2) partition formatted with btrfs on top.
The system has 4 cores.

Steps for reproducing the issue
For me it happens randomly for 1 kernel thread, then after a while 3 others also freeze and the system is then dead.
Starting a heavy io task like btrfs scrub will reproduce the issue within seconds.

eg first after a few minutes with heavy io
ps aux
...
root 265 0.5 0.0 0 0 ? D 21:16 0:02 [kworker/u8:3+kcryptd/253:2]
...

then a bit later
root 265 0.1 0.0 0 0 ? D 21:16 0:02 [kworker/u8:3+kcryptd/253:2]
root 310 0.1 0.0 0 0 ? D 21:16 0:02 [kworker/u8:8+kcryptd/253:2]
root 312 0.2 0.0 0 0 ? D 21:16 0:03 [kworker/u8:10+btrfs-endio-write]
root 313 0.1 0.0 0 0 ? D 21:16 0:02 [kworker/u8:11+btrfs-endio-write]
root 314 0.1 0.0 0 0 ? D 21:16 0:01 [kworker/u8:12+flush-btrfs-2]
root 315 0.1 0.0 0 0 ? D 21:16 0:02 [kworker/u8:13+kcryptd/253:2]
root 1612 0.0 0.0 0 0 ? D 21:16 0:00 [btrfs-transacti]
root 4133 0.0 0.0 0 0 ? D 21:22 0:00 [kworker/u8:0+kcryptd/253:2]
root 4135 0.0 0.0 0 0 ? D 21:22 0:00 [kworker/u8:4+btrfs-endio-write]
root 4137 0.0 0.0 0 0 ? D 21:22 0:00 [kworker/u8:14+btrfs-endio-write]
root 4977 0.0 0.0 0 0 ? D 21:30 0:00 [kworker/u8:1+btrfs-endio-write]
root 4978 0.0 0.0 0 0 ? D 21:30 0:00 [kworker/u8:2+btrfs-endio-write]

Additional info
Ubuntu focal install with btrfs on luks partitions.
Kernel info
Linux 5.4.0-31-generic #35-Ubuntu SMP x86_64 x86_64 x86_64 GNU/Linux

lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 39 bits physical, 48 bits virtual
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 95
Model name: Intel(R) Atom(TM) CPU C3558 @ 2.20GHz
Stepping: 1
CPU MHz: 2200.413
CPU max MHz: 2200,0000
CPU min MHz: 800,0000
BogoMIPS: 4400.00
Virtualization: VT-x
L1d cache: 96 KiB
L1i cache: 128 KiB
L2 cache: 8 MiB
NUMA node0 CPU(s): 0-3
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP disabled, RSB filling
Vulnerability Tsx async abort: Not affected
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc
art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg cx16 xtpr
pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave rdrand lahf_lm 3dnowprefetch cpuid_fault epb cat_l2 ssbd ibrs ibpb stibp tpr_shadow vnmi flex
priority ept vpid ept_ad fsgsbase tsc_adjust smep erms mpx rdt_a rdseed smap clflushopt intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves dtherm arat pln pts md_clea
r arch_capabilities

kernel crypto flags
cat /boot/config-5.4.0-31-generic | grep AES
CONFIG_CRYPTO_AEGIS128_AESNI_SSE2=m
CONFIG_CRYPTO_LIB_AES=y
CONFIG_CRYPTO_AES=y
CONFIG_CRYPTO_AES_TI=m
CONFIG_CRYPTO_AES_NI_INTEL=m
CONFIG_CRYPTO_CAMELLIA_AESNI_AVX_X86_64=m
CONFIG_CRYPTO_CAMELLIA_AESNI_AVX2_X86_64=m
CONFIG_CRYPTO_DEV_PADLOCK_AES=m

/etc/crypttab
LUKS_DATA_6T_1 UUID=the_uuid /etc/....keyfile luks

Related bug reports:
https://gitlab.com/cryptsetup/cryptsetup/-/issues/568 -> rejected, referred to ubuntu kernel team.
---
ProblemType: Bug
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Mar 31 10:18 seq
 crw-rw---- 1 root audio 116, 33 Mar 31 10:18 timer
AplayDevices: aplay: device_list:274: no soundcards found...
ApportVersion: 2.20.11-0ubuntu27.16
Architecture: amd64
ArecordDevices: arecord: device_list:274: no soundcards found...
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CasperMD5CheckResult: skip
DistroRelease: Ubuntu 20.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: Supermicro Super Server
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_DK.UTF-8
 SHELL=/bin/bash
ProcFB: 0 astdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-5.8.0-48-generic root=UUID=6612e070-1987-43a4-86f6-1388915f48eb ro rootflags=subvol=@ quiet splash vt.handoff=7
ProcVersionSignature: Ubuntu 5.8.0-48.54~20.04.1-generic 5.8.18
RelatedPackageVersions:
 linux-restricted-modules-5.8.0-48-generic N/A
 linux-backports-modules-5.8.0-48-generic N/A
 linux-firmware 1.187.10
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: focal
Uname: Linux 5.8.0-48-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 11/05/2019
dmi.bios.release: 5.13
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 1.2
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: A2SDi-4C-HLN4F
dmi.board.vendor: Supermicro
dmi.board.version: 1.00
dmi.chassis.asset.tag: To be filled by O.E.M.
dmi.chassis.type: 17
dmi.chassis.vendor: Supermicro
dmi.chassis.version: 0123456789
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1.2:bd11/05/2019:br5.13:svnSupermicro:pnSuperServer:pvr0123456789:rvnSupermicro:rnA2SDi-4C-HLN4F:rvr1.00:cvnSupermicro:ct17:cvr0123456789:
dmi.product.family: To be filled by O.E.M.
dmi.product.name: Super Server
dmi.product.sku: To be filled by O.E.M.
dmi.product.version: 0123456789
dmi.sys.vendor: Supermicro

Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1922022

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
tags: added: focal
Revision history for this message
Jesper Utoft (jesper-utoft) wrote : CRDA.txt

apport information

tags: added: apport-collected
description: updated
Revision history for this message
Jesper Utoft (jesper-utoft) wrote : CurrentDmesg.txt

apport information

Revision history for this message
Jesper Utoft (jesper-utoft) wrote : Lspci.txt

apport information

Revision history for this message
Jesper Utoft (jesper-utoft) wrote : Lspci-vt.txt

apport information

Revision history for this message
Jesper Utoft (jesper-utoft) wrote : Lsusb.txt

apport information

Revision history for this message
Jesper Utoft (jesper-utoft) wrote : Lsusb-t.txt

apport information

Revision history for this message
Jesper Utoft (jesper-utoft) wrote : Lsusb-v.txt

apport information

Revision history for this message
Jesper Utoft (jesper-utoft) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
Jesper Utoft (jesper-utoft) wrote : ProcCpuinfoMinimal.txt

apport information

Revision history for this message
Jesper Utoft (jesper-utoft) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Jesper Utoft (jesper-utoft) wrote : ProcModules.txt

apport information

Revision history for this message
Jesper Utoft (jesper-utoft) wrote : UdevDb.txt

apport information

Revision history for this message
Jesper Utoft (jesper-utoft) wrote : WifiSyslog.txt

apport information

Revision history for this message
Jesper Utoft (jesper-utoft) wrote : acpidump.txt

apport information

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
description: updated
description: updated
summary: - Btrfs blocked io when running on top of luks encrypted partition
+ Btrfs blocked/frozen io when running on top of luks encrypted partition
tags: added: btrfs kernel-fs luks
Revision history for this message
Jesper Utoft (jesper-utoft) wrote :

I have tested additional kernels from https://kernel.ubuntu.com/~kernel-ppa/mainline/
5.9.16-050916 generic-amd64 -> affected by bug
5
.10.28-051028 generic-amd64 -> NOT affected by bug
5
.11.12-051112 generic-amd64 -> NOT affected by bug

Revision history for this message
Jesper Utoft (jesper-utoft) wrote :

As Ubuntu 20.04.3 LTS is expected to run kernel 5.11.? i will close this issue as it will be fixed by that upgrade.

affects: linux (Ubuntu) → linux-hwe-5.8 (Ubuntu)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.