ubuntu_zram_smoke test will cause soft lockup on Artful ThunderX ARM64

This change was made by a bot.

How do i reproduce this?

git clone git://kernel.ubuntu.com/ubuntu/autotest
git clone git://kernel.ubuntu.com/ubuntu/autotest-client-tests
cd autotest/client
rmdir tests
ln -s ../../autotest-client-tests tests
cd ../..
sudo autotest/client/autotest-local autotest/client/tests/ubuntu_zram_smoke_test/control

It's an upstream bug - i can reproduce it on lundmark (thunderx) with 4.14.26 (latest 4.14 stable release) using the artful .config (make oldconfig and accept all defaults values - i'll attach the config below).

It appears it only shows up on ThunderX SOCs (the same test on a hisilicon d05 worked fine), and the toolchain doesn't matter, it shows up when the kernel was compiled with the artful or xenial toolchain, no matter what.

I'm reducing the scope before reporting it upstream.

ubuntu@lundmark:~$ uname -a
Linux lundmark 4.14.26+ #3 SMP Mon Mar 12 15:48:47 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux

ubuntu@lundmark:~$ sudo ./zram_smoke_test.sh
PASSED: zram module loaded
PASSED: got 48 compression streams
PASSED: got lzo lz4 deflate lz4hc 842 compression algorithmns
PASSED: mkfs.ext4 on ZRAM successful
PASSED: ext4 on ZRAM (lzo), memory used: 41945534 4497
PASSED: mkfs.ext4 on ZRAM successful
PASSED: ext4 on ZRAM (lz4), memory used: 41945256 4006

[from the serial console]
dmesg:
...
[ 289.708789] zram: Added device: zram0
[ 291.743271] zram0: detected capacity change from 0 to 134217728
[ 291.909193] EXT4-fs (zram0): mounted filesystem with ordered data mode. Opts: (null)
[ 295.177609] zram0: detected capacity change from 134217728 to 0
[ 296.291895] zram0: detected capacity change from 0 to 134217728
[ 296.394257] EXT4-fs (zram0): mounted filesystem with ordered data mode. Opts: (null)
[ 299.602110] zram0: detected capacity change from 134217728 to 0
[ 300.606078] zram0: detected capacity change from 0 to 134217728
[ 327.694207] watchdog: BUG: soft lockup - CPU#25 stuck for 23s! [mkfs.ext4:2016]
[ 327.701593] Modules linked in: lz4 lz4_compress zram nicvf nicpf nls_iso8859_1 ast ttm thunder_bgx drm_kms_helper thunder_xcv thunderx_mmc mdio_thund
er mdio_cavium drm fb_sys_fops syscopyarea sysfillrect sysimgblt i2c_thunderx i2c_smbus i2c_algo_bit thunderx_edac thunderx_zip cavium_rng_vf shpchp cav
ium_rng aes_ce_blk crypto_simd cryptd aes_ce_cipher crc32_ce crct10dif_ce ghash_ce ipmi_ssif aes_arm64 ipmi_devintf gpio_keys sha2_ce sha256_arm64 sha1_
ce ipmi_msghandler uio_pdrv_genirq uio ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autof
s4
[ 327.701693] CPU: 25 PID: 2016 Comm: mkfs.ext4 Not tainted 4.14.26+ #3
[ 327.701695] Hardware name: Cavium ThunderX CRB/To be filled by O.E.M., BIOS 5.11 12/12/2012
[ 327.701698] task: ffff801f56dc0f00 task.stack: ffff000012768000
[ 327.701711] PC is at zip_deflate+0x180/0x2e8 [thunderx_zip]
[ 327.701717] LR is at zip_deflate+0x17c/0x2e8 [thunderx_zip]
[ 327.701720] pc : [<ffff000002031618>] lr : [<ffff000002031614>] pstate: 20400145
[ 327.701721] sp : ffff00001276b570
[ 327.701723] x29: ffff00001276b570 x28: 0000000000000000
[ 327.701728] x27: ffff801f27f2ec00 x26: 0000000000000000
[ 327.701733] x25: ffff801f58b5c000 x24: ffff801f6f72ac18
[ 327.701738] x23: 0000000000001000 x22: ffff00001276b678
[ 327.701743] x21: ffff801f59214880 x20: ffff801f6f72ac18
[ 327.701748] x19: ffff00001276b5f8 x18: 0000000000000000
[ 327.701752] x17: 0000000000000001 x16: 0000000000000000
[ 327.701757] x15: 0000aaab222018db x14: 00...

One thing to note, this issue can be reproduced with the Xenial HWE kernel on ThunderX systems as well. (Seems ok with Moonshot system)

Tested with a Moonshot system, this ubuntu_zram_smoke test can pass with Artful kernel.

So this matches ppisati's finding in comment #5

Ok, here is what i found so far:

the problem lies in the 'thunderx_zip' driver, that is the driver for the hw accelerated zip compressor / decompressor ip block - it kicks in once we select the deflate method for the zram device.

How way to reproduce it:

# modprobe zram
# echo 1 > /sys/block/zram0/reset
# echo deflate > /sys/block/zram0/comp_algorithm
# echo 128M > /sys/block/zram0/disksize
# mkfs.ext4 -F /dev/zram0
[stuck forever here]

Two trivial workarounds:

-blacklist the thunderx_zip kmod:

# rmmod thunderx_zip
# echo 'blacklist thunderx_zip' >> /etc/modprobe.d/blacklist.conf

or

-disable the CRYPTO_DEV_CAVIUM_ZIP kconfig (and void building thunderx_zip kmod) and recompile

As to what is causing it, till yesterday, it appeared as the problem was connected to the arm64 kpti but now i'm sure it is not:

the problem started to appear in 4.13.0-37-generic #42, while 4.13.0-36-generic #40 was immune and the only difference between those two kernels is the arm64 kpti patchset.
You can easily reproduce it in the upstream stable/linux-4.14.y tree too (4.14.26 is affected for example).

$ make defconfig
$ echo "CONFIG_CRYPTO_DEV_CAVIUM_ZIP=m" >> .config
$ make oldconfig

build and install as usual, and then try the reproducer above.

But what i found this morning, is that even the original 4.14.0 release is affected, but that release clearly doesn't contain the kpti patches.

Now what i want to try is:

1) test it on different hardware (one thing that i noticed is that if the thunderx_zip kmod is loaded at boot or later in the board life cycle, that slightly changes the error and that smells a lot like memory corruption)
2) test it with 4.15x and 4.16

I'll write another update when i have more data.

Ok, i tested it on another board (seuss) with 4.14.27, 4.15.10 and 4.16-rc4:

4.14.0: affected
4.14.27: affected
4.15.10: affected
4.16-rc: doesn't boot at all, so i can't tell

so we can rule out the bad hardware variable, and we can say for sure that is not related to kpti at all.

Another thing: once building a kernel from upstream, you clearly need to enable ZSMALLOC and ZRAM too.

I can reproduce this issue with 4.13.0-37-generic. The debugfs info for the zip driver shows that 2 requests are pending. Either the requests are not submitted properly (Decomp Req Submitted=0) or
not completed (DOORBELL regs = 0).

root@crb1s:/sys/kernel/debug/thunderx_zip# cat zip_stats
        ZIP Device 0 Stats
-----------------------------------
Comp Req Submitted : 0
Comp Req Completed : 0
Compress In Bytes : 0
Compressed Out Bytes : 0
Average Chunk size : 0
Average Compression ratio : 0
Decomp Req Submitted : 0
Decomp Req Completed : 0
Decompress In Bytes : 0
Decompressed Out Bytes : 0
Decompress Bad requests : 0
Pending Req : 2
---------------------------------
root@crb1s:/sys/kernel/debug/thunderx_zip# cat zip_regs
--------------------------------
     ZIP Device 0 Registers
--------------------------------
ZIP_CMD_CTL : 0x0000000000000002
ZIP_THROTTLE : 0x0000000000000010
ZIP_CONSTANTS : 0x02017c0020060000
ZIP_QUE0_MAP : 0x0000000000000003
ZIP_QUE1_MAP : 0x0000000000000003
ZIP_QUE_ENA : 0x0000000000000003
ZIP_QUE_PRI : 0x0000000000000003
ZIP_QUE0_DONE : 0x0000000000000000
ZIP_QUE1_DONE : 0x0000000000000000
ZIP_QUE0_DOORBELL : 0x0000000000000000
ZIP_QUE1_DOORBELL : 0x0000000000000000
ZIP_QUE0_SBUF_ADDR : 0x00000000fa040100
ZIP_QUE1_SBUF_ADDR : 0x00000000fa042000
ZIP_QUE0_SBUF_CTL : 0x000003f100000000
ZIP_QUE1_SBUF_CTL : 0x000003f100000000

This is a regression caused by CONFIG_VMAP_STACK. The driver uses __pa (phys_to_virt)
on a stack address which does not work with virtual mapped stacks.

To solve this upstream will require a bit more work, I'm attaching a minimal patch
to work-around by using kmalloc for the problematic allocation.

Can be reproduced with X-hwe on ThunderX as well.

Patches posted:
https://marc.info/?l=linux-kernel&m=152224242828223&w=2
https://marc.info/?l=linux-kernel&m=152224242228218&w=2

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-artful' to 'verification-done-artful'. If the problem still exists, change the tag 'verification-needed-artful' to 'verification-failed-artful'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug was fixed in the package linux - 4.15.0-15.16

---------------
linux (4.15.0-15.16) bionic; urgency=medium

  * linux: 4.15.0-15.16 -proposed tracker (LP: #1761177)

  * FFe: Enable configuring resume offset via sysfs (LP: #1760106)
    - PM / hibernate: Make passing hibernate offsets more friendly

  * /dev/bcache/by-uuid links not created after reboot (LP: #1729145)
    - SAUCE: (no-up) bcache: decouple emitting a cached_dev CHANGE uevent

  * Ubuntu18.04:POWER9:DD2.2 - Unable to start a KVM guest with default machine
    type(pseries-bionic) complaining "KVM implementation does not support
    Transactional Memory, try cap-htm=off" (kvm) (LP: #1752026)
    - powerpc: Use feature bit for RTC presence rather than timebase presence
    - powerpc: Book E: Remove unused CPU_FTR_L2CSR bit
    - powerpc: Free up CPU feature bits on 64-bit machines
    - powerpc: Add CPU feature bits for TM bug workarounds on POWER9 v2.2
    - powerpc/powernv: Provide a way to force a core into SMT4 mode
    - KVM: PPC: Book3S HV: Work around transactional memory bugs in POWER9
    - KVM: PPC: Book3S HV: Work around XER[SO] bug in fake suspend mode
    - KVM: PPC: Book3S HV: Work around TEXASR bug in fake suspend state

  * Important Kernel fixes to be backported for Power9 (kvm) (LP: #1758910)
    - powerpc/mm: Fixup tlbie vs store ordering issue on POWER9

  * Ubuntu 18.04 - IO Hang on some namespaces when running HTX with 16
    namespaces (Bolt / NVMe) (LP: #1757497)
    - powerpc/64s: Fix lost pending interrupt due to race causing lost update to
      irq_happened

  * fwts-efi-runtime-dkms 18.03.00-0ubuntu1: fwts-efi-runtime-dkms kernel module
    failed to build (LP: #1760876)
    - [Packaging] include the retpoline extractor in the headers

linux (4.15.0-14.15) bionic; urgency=medium

  * linux: 4.15.0-14.15 -proposed tracker (LP: #1760678)

  * [Bionic] mlx4 ETH - mlnx_qos failed when set some TC to vendor
    (LP: #1758662)
    - net/mlx4_en: Change default QoS settings

  * AT_BASE_PLATFORM in AUXV is absent on kernels available on Ubuntu 17.10
    (LP: #1759312)
    - powerpc/64s: Fix NULL AT_BASE_PLATFORM when using DT CPU features

  * Bionic update to 4.15.15 stable release (LP: #1760585)
    - net: dsa: Fix dsa_is_user_port() test inversion
    - openvswitch: meter: fix the incorrect calculation of max delta_t
    - qed: Fix MPA unalign flow in case header is split across two packets.
    - tcp: purge write queue upon aborting the connection
    - qed: Fix non TCP packets should be dropped on iWARP ll2 connection
    - sysfs: symlink: export sysfs_create_link_nowarn()
    - net: phy: relax error checking when creating sysfs link netdev->phydev
    - devlink: Remove redundant free on error path
    - macvlan: filter out unsupported feature flags
    - net: ipv6: keep sk status consistent after datagram connect failure
    - ipv6: old_dport should be a __be16 in __ip6_datagram_connect()
    - ipv6: sr: fix NULL pointer dereference when setting encap source address
    - ipv6: sr: fix scheduling in RCU when creating seg6 lwtunnel state
    - mlxsw: spectrum_buffers: Set a minimum quota for CPU port traffic
    - net: phy: Tell caller result ...

-- before proposed --
Ubuntu 17.10 (GNU/Linux 4.13.0-38-generic aarch64)
ubuntu@anuchin:~$ sudo bash
root@anuchin:~# modprobe zram
root@anuchin:~# echo 1 > /sys/block/zram0/reset
root@anuchin:~# echo deflate > /sys/block/zram0/comp_algorithm
root@anuchin:~# echo 128M > /sys/block/zram0/disksize
root@anuchin:~# mkfs.ext4 -F /dev/zram0
mke2fs 1.43.5 (04-Aug-2017)
Discarding device blocks: done
Creating filesystem with 32768 4k blocks and 32768 inodes

Allocating group tables: done
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information:

manjo@lazy:~$ ipmitool -H 10.229.48.14 -I lanplus -U admin -P admin sol activate
[SOL Session operational. Use ~? for help]
[ 1420.012055] watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [mkfs.ext4:3121]
[ 1424.023938] watchdog: BUG: soft lockup - CPU#40 stuck for 23s! [kworker/u96:2:406]
[ 1454.099071] INFO: rcu_sched self-detected stall on CPU
[ 1454.104260] 14-...: (14998 ticks this GP) idle=796/140000000000001/0 softirq=4896/4896 fqs=7500
[ 1454.107070] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 1454.107076] 14-...: (14998 ticks this GP) idle=796/140000000000001/0 softirq=4896/4896 fqs=7500
[ 1454.107077] (detected by 40, t=15002 jiffies, g=3929, c=3928, q=4597)
[ 1454.134272] (t=15008 jiffies g=3929 c=3928 q=4597)

-- after proposed --
ubuntu@anuchin:~$ uname -a
Linux anuchin 4.13.0-39-generic #44-Ubuntu SMP Thu Apr 5 14:20:13 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux
ubuntu@anuchin:~$ sudo bash
root@anuchin:~# modprobe zram
root@anuchin:~# echo 1 > /sys/block/zram0/reset
root@anuchin:~# echo deflate > /sys/block/zram0/comp_algorithm
root@anuchin:~# echo 128M > /sys/block/zram0/disksize
root@anuchin:~# mkfs.ext4 -F /dev/zram0
mke2fs 1.43.5 (04-Aug-2017)
Discarding device blocks: done
Creating filesystem with 32768 4k blocks and 32768 inodes

Allocating group tables: done
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

root@anuchin:~#

This bug was fixed in the package linux - 4.13.0-39.44

---------------
linux (4.13.0-39.44) artful; urgency=medium

  * linux: 4.13.0-39.44 -proposed tracker (LP: #1761456)

  * intel-microcode 3.20180312.0 causes lockup at login screen(w/ linux-
    image-4.13.0-37-generic) (LP: #1759920) // CVE-2017-5715 (Spectre v2
    Intel) // CVE-2017-5754
    - x86/mm: Reinitialize TLB state on hotplug and resume

  * intel-microcode 3.20180312.0 causes lockup at login screen(w/ linux-
    image-4.13.0-37-generic) (LP: #1759920) // CVE-2017-5715 (Spectre v2 Intel)
    - Revert "x86/mm: Only set IBPB when the new thread cannot ptrace current
      thread"
    - x86/speculation: Use Indirect Branch Prediction Barrier in context switch

  * DKMS driver builds fail with: Cannot use CONFIG_STACK_VALIDATION=y, please
    install libelf-dev, libelf-devel or elfutils-libelf-devel (LP: #1760876)
    - [Packaging] include the retpoline extractor in the headers

  * retpoline hints: primary infrastructure and initial hints (LP: #1758856)
    - [Packaging] retpoline-extract: flag *0xNNN(%reg) branches
    - x86/speculation, objtool: Annotate indirect calls/jumps for objtool
    - x86/speculation, objtool: Annotate indirect calls/jumps for objtool on 32bit
    - x86/paravirt, objtool: Annotate indirect calls
    - [Packaging] retpoline -- add safe usage hint support
    - [Packaging] retpoline-check -- only report additions
    - [Packaging] retpoline -- widen indirect call/jmp detection
    - [Packaging] retpoline -- elide %rip relative indirections
    - [Packaging] retpoline -- clear hint information from packages
    - KVM: x86: Make indirect calls in emulator speculation safe
    - KVM: VMX: Make indirect call speculation safe
    - x86/boot, objtool: Annotate indirect jump in secondary_startup_64()
    - SAUCE: early/late -- annotate indirect calls in early/late initialisation
      code
    - SAUCE: vga_set_mode -- avoid jump tables
    - [Config] retpoline -- switch to new format
    - [Packaging] retpoline hints -- handle missing files when RETPOLINE not
      enabled
    - [Packaging] final-checks -- remove check for empty retpoline files

  * retpoline: ignore %cs:0xNNN constant indirections (LP: #1752655)
    - [Packaging] retpoline -- elide %cs:0xNNNN constants on i386

  * zfs system process hung on container stop/delete (LP: #1754584)
    - SAUCE: Fix non-prefaulted page deadlock (LP: #1754584)

  * zfs-linux 0.6.5.11-1ubuntu5 ADT test failure with linux 4.15.0-1.2
    (LP: #1737761)
    - SAUCE: (noup) Update zfs to 0.6.5.11-1ubuntu3.2

  * AT_BASE_PLATFORM in AUXV is absent on kernels available on Ubuntu 17.10
    (LP: #1759312)
    - powerpc/64s: Fix NULL AT_BASE_PLATFORM when using DT CPU features

  * btrfs and tar sparse truncate archives (LP: #1757565)
    - Btrfs: move definition of the function btrfs_find_new_delalloc_bytes
    - Btrfs: fix reported number of inode blocks after buffered append writes

  * efifb broken on ThunderX-based Gigabyte nodes (LP: #1758375)
    - drivers/fbdev/efifb: Allow BAR to be moved instead of claiming it

  * Intel i40e PF reset due to incorrect MDD detection (continues...)
    (LP: #1723127)
    - i40e/i40ev...

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug was erroneously marked for verification in bionic; verification is not required and verification-needed-bionic is being removed.