zfs system process hung on container stop/delete

This change was made by a bot.

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.16 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16-rc4

> Did this issue start happening after an update/upgrade?
> Was there a prior kernel version where you were not having this particular problem?

This was the first time I have attempted this so I do not know about previous kernels.

After installing 4.16-rc4 I could not re-attempt to reproduce because the zfs modules are not present:
modprobe: FATAL: Module zfs not found in directory /lib/modules/4.16.0-041600rc4-generic

I changed lxd to use btrfs instead of zfs and attempted to reproduce and was not able to.

Since I need zfs I did mark as kernel-fixed-upstream.

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

This also occurs with the current proposed kernel in bionic when I used lxd + zfs:

Linux x1 4.15.0-12-generic #13-Ubuntu SMP Thu Mar 8 06:24:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

I did an apport-collect 1754584 with that kernel after reproducing.

Reproduced again on 4.13 with zfs:

Linux x1 4.13.0-32-generic #35-Ubuntu SMP Thu Jan 25 09:13:46 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

zfs info:

filename: /lib/modules/4.13.0-32-generic/kernel/zfs/zfs/zfs.ko
version: 0.6.5.11-1ubuntu3
license: CDDL
author: OpenZFS on Linux
description: ZFS
srcversion: 2F9D27546C1D6992FF66D5C
depends: spl,znvpair,zcommon,zunicode,zavl
name: zfs
vermagic: 4.13.0-32-generic SMP mod_unload

Would it be possible for you to see if changing lxd to use btrfs instead of zfs on Bionic causes the bug?

> Would it be possible for you to see if changing lxd
> to use btrfs instead of zfs on Bionic causes the bug?

Proposed Bionic kernel (4.15.0-12) + btrfs PASSED

Results
---
4.16-rc4 + btrfs - PASSED
4.15.0-12 + btrfs - PASSED
4.15.0-12 + zfs - FAILS
4.15.0-10 + zfs - FAILS
4.13.0-32 + zfs - FAILS

4.15.0-12 zfs info:

filename: /lib/modules/4.15.0-12-generic/kernel/zfs/zfs/zfs.ko
version: 0.7.5-1ubuntu1
license: CDDL
author: OpenZFS on Linux
description: ZFS
srcversion: 9C78552EF2E79ADAE6389FB

This does recreate on a zfs filesystem also. No lxc is required.
Recreated as root and non-root on 4.15.0-10-generic .

recreate in a vm that had a free block device /dev/vdb.

sudo zpool create -f myzfs /dev/vdb
sudo zfs create -p myzfs/myproject/foo
sudo chmod 1777 /myzfs/myproject/foo

cd /myzfs/myproject/foo

apt-get update
apt-get install -y ubuntu-dev-tools
apt-get build-dep -y libaio
pull-lp-source libaio
sh -c "cd libaio-*/ && dpkg-buildpackage -j4 -us -uc"

I marked this private. Unpriviledged user on a zfs filesystem can essentially create processes that cannot be killed.

This can be reproduced with 4.13.0-37-generic, 0.6.5.11-1ubuntu3.2 so it's not a regression.

And also on 4.4.0-116-generic, zfs 0.6.5.6-0ubuntu19

Boiled it down to a minimal async I/O reproducer as follows:

#include <stdio.h>
#include <unistd.h>
#include <sys/mman.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <libaio.h>
#include <fcntl.h>
#include <err.h>

io_context_t io_ctx;

void do_sync_io(struct iocb *iocb)
{
        struct io_event event;
        struct iocb *iocbs[] = { iocb };
        struct timespec ts = { 30, 0 };

        if (io_submit(io_ctx, 1, iocbs) != 1)
                err(1, "io_submit failed");
        if (io_getevents(io_ctx, 0, 1, &event, &ts) != 1)
                err(1, "io_getevents failed");
}

int main(void)
{
        char *buf;
        int rwfd, status = 0, res, page_size = getpagesize();
        struct iocb iocb;

        if (io_queue_init(1024, &io_ctx))
                err(1, "io_queue_init failed");
        rwfd = open("testfile", O_RDWR | O_CREAT);
        if (rwfd < 0)
                err(1, "open failed");
        if (ftruncate(rwfd, 512) < 0)
                err(1, "ftruncate failed");
        buf = mmap(0, page_size, PROT_READ | PROT_WRITE, MAP_SHARED, rwfd, 0);
        if (buf == MAP_FAILED)
                err(1, "mmap failed");

        (void)io_prep_pwrite(&iocb, rwfd, buf, 512, 0);
        do_sync_io(&iocb);

        (void)io_prep_pread(&iocb, rwfd, buf, 512, 0);
        do_sync_io(&iocb);
}

And still occurs with the latest ZFS from today's SPL+ZFS git repos.

I suspect this happens because the page is being locked in mappedread and again in filemap_page_mkwrite. Since the page is already locked, the locking of the page in filemap_page_mkwrite causes the task to block indefinitely.

Touching page before the pread causes the page to be dirty and a page fault occurs and then the code does not hang.

Upstream have developed a fix for this, I'll get it packaged and tested on Monday: https://github.com/zfsonlinux/zfs/pull/7339

This bug was fixed in the package zfs-linux - 0.7.5-1ubuntu11

---------------
zfs-linux (0.7.5-1ubuntu11) bionic; urgency=medium

  * Fix mmap'd libaio read on non-prefaulted page deadlock (LP: #1754584)
    - ensure page is pre-faulted in to avoid deadlock

 -- Colin Ian King <email address hidden> Mon, 26 Mar 2018 09:29:23 +0100

Hello Joshua, or anyone else affected,

Accepted zfs-linux into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zfs-linux/0.6.5.11-1ubuntu3.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Joshua, or anyone else affected,

Accepted zfs-linux into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zfs-linux/0.6.5.6-0ubuntu20 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I am marking this verification-failed as my steps to reproduce this and the ones in step #24 still fail in Bionic, Artful, and Xenial.

Bionic: 0.7.5-1ubuntu12
https://paste.ubuntu.com/p/mWVnhdFpRp/

Artful: 0.6.5.11-1ubuntu3.3
https://paste.ubuntu.com/p/RQmxNS7Vxn/

Xenial: 0.6.5.6-0ubuntu20
https://paste.ubuntu.com/p/fZpfWKyYMk/

@colin - is there a requirement to have a newer kernel and not just a newer zfs-linux for this to be fixed?

One requires both the user space and the kernel packages to be installed, so please re-test when the kernel also appears in -proposed. Apologies for the confusion.

Is there a kernel ready for testing this on bionic or xenial?

Not yet

This bug was fixed in the package linux - 4.15.0-15.16

---------------
linux (4.15.0-15.16) bionic; urgency=medium

  * linux: 4.15.0-15.16 -proposed tracker (LP: #1761177)

  * FFe: Enable configuring resume offset via sysfs (LP: #1760106)
    - PM / hibernate: Make passing hibernate offsets more friendly

  * /dev/bcache/by-uuid links not created after reboot (LP: #1729145)
    - SAUCE: (no-up) bcache: decouple emitting a cached_dev CHANGE uevent

  * Ubuntu18.04:POWER9:DD2.2 - Unable to start a KVM guest with default machine
    type(pseries-bionic) complaining "KVM implementation does not support
    Transactional Memory, try cap-htm=off" (kvm) (LP: #1752026)
    - powerpc: Use feature bit for RTC presence rather than timebase presence
    - powerpc: Book E: Remove unused CPU_FTR_L2CSR bit
    - powerpc: Free up CPU feature bits on 64-bit machines
    - powerpc: Add CPU feature bits for TM bug workarounds on POWER9 v2.2
    - powerpc/powernv: Provide a way to force a core into SMT4 mode
    - KVM: PPC: Book3S HV: Work around transactional memory bugs in POWER9
    - KVM: PPC: Book3S HV: Work around XER[SO] bug in fake suspend mode
    - KVM: PPC: Book3S HV: Work around TEXASR bug in fake suspend state

  * Important Kernel fixes to be backported for Power9 (kvm) (LP: #1758910)
    - powerpc/mm: Fixup tlbie vs store ordering issue on POWER9

  * Ubuntu 18.04 - IO Hang on some namespaces when running HTX with 16
    namespaces (Bolt / NVMe) (LP: #1757497)
    - powerpc/64s: Fix lost pending interrupt due to race causing lost update to
      irq_happened

  * fwts-efi-runtime-dkms 18.03.00-0ubuntu1: fwts-efi-runtime-dkms kernel module
    failed to build (LP: #1760876)
    - [Packaging] include the retpoline extractor in the headers

linux (4.15.0-14.15) bionic; urgency=medium

  * linux: 4.15.0-14.15 -proposed tracker (LP: #1760678)

  * [Bionic] mlx4 ETH - mlnx_qos failed when set some TC to vendor
    (LP: #1758662)
    - net/mlx4_en: Change default QoS settings

  * AT_BASE_PLATFORM in AUXV is absent on kernels available on Ubuntu 17.10
    (LP: #1759312)
    - powerpc/64s: Fix NULL AT_BASE_PLATFORM when using DT CPU features

  * Bionic update to 4.15.15 stable release (LP: #1760585)
    - net: dsa: Fix dsa_is_user_port() test inversion
    - openvswitch: meter: fix the incorrect calculation of max delta_t
    - qed: Fix MPA unalign flow in case header is split across two packets.
    - tcp: purge write queue upon aborting the connection
    - qed: Fix non TCP packets should be dropped on iWARP ll2 connection
    - sysfs: symlink: export sysfs_create_link_nowarn()
    - net: phy: relax error checking when creating sysfs link netdev->phydev
    - devlink: Remove redundant free on error path
    - macvlan: filter out unsupported feature flags
    - net: ipv6: keep sk status consistent after datagram connect failure
    - ipv6: old_dport should be a __be16 in __ip6_datagram_connect()
    - ipv6: sr: fix NULL pointer dereference when setting encap source address
    - ipv6: sr: fix scheduling in RCU when creating seg6 lwtunnel state
    - mlxsw: spectrum_buffers: Set a minimum quota for CPU port traffic
    - net: phy: Tell caller result ...

Hi @powersj,

The Xenial and Artful kernels currently in -proposed have the fix for this issue.

Using two NUCs, one with Xenial and one with Artful I was able to verify both the kernel and zfs fixes. Marking verification-done-artful and erification-done-xenial

Artful log: https://paste.ubuntu.com/p/jjZWTNqPjS/

Artful versions:
zfsutils-linux: 0.6.5.6-0ubuntu20
linux-signed-generic: 4.4.0.121.127

Xenial log: https://paste.ubuntu.com/p/b6gvKbpgWC/

Xenial versions:
zfsutils-linux: 0.6.5.11-1ubuntu3.3
linux-signed-generic: 4.13.0.39.42

Thanks for verifying Joshua. Much appreciated!

This bug was fixed in the package zfs-linux - 0.6.5.6-0ubuntu20

---------------
zfs-linux (0.6.5.6-0ubuntu20) xenial; urgency=medium

  * Fix mmap'd libaio read on non-prefaulted page deadlock (LP: #1754584)
    - Revert previous fix as there still is a potential for deadlock
      on the prefaulted page(s). Resolve the issue by only taking a
      reference on the page when copying it and not the page lock.
      The inode range lock protects against concurrent updates via
      zfs_read() and zfs_write().

 -- Colin Ian King <email address hidden> Wed, 28 Mar 2018 14:35:11 +0100

The verification of the Stable Release Update for zfs-linux has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package zfs-linux - 0.6.5.11-1ubuntu3.3

---------------
zfs-linux (0.6.5.11-1ubuntu3.3) artful; urgency=medium

  * Fix mmap'd libaio read on non-prefaulted page deadlock (LP: #1754584)
    - Revert previous fix as there still is a potential for deadlock
      on the prefaulted page(s). Resolve the issue by only taking a
      reference on the page when copying it and not the page lock.
      The inode range lock protects against concurrent updates via
      zfs_read() and zfs_write().

 -- Colin Ian King <email address hidden> Wed, 28 Mar 2018 08:56:18 +0100

This bug was fixed in the package linux - 4.4.0-121.145

---------------
linux (4.4.0-121.145) xenial; urgency=medium

  * linux: 4.4.0-121.145 -proposed tracker (LP: #1763687)

  * Ubuntu-4.4.0-120.144 fails to boot on arm64* hardware (LP: #1763644)
    - [Config] arm64: disable BPF_JIT_ALWAYS_ON

linux (4.4.0-120.144) xenial; urgency=medium

  * linux: 4.4.0-120.144 -proposed tracker (LP: #1761438)

  * intel-microcode 3.20180312.0 causes lockup at login screen(w/ linux-
    image-4.13.0-37-generic) (LP: #1759920) // CVE-2017-5715 (Spectre v2 Intel)
    - Revert "x86/mm: Only set IBPB when the new thread cannot ptrace current
      thread"
    - x86/speculation: Use Indirect Branch Prediction Barrier in context switch

  * DKMS driver builds fail with: Cannot use CONFIG_STACK_VALIDATION=y, please
    install libelf-dev, libelf-devel or elfutils-libelf-devel (LP: #1760876)
    - [Packaging] include the retpoline extractor in the headers

  * retpoline hints: primary infrastructure and initial hints (LP: #1758856)
    - [Packaging] retpoline-extract: flag *0xNNN(%reg) branches
    - x86/speculation, objtool: Annotate indirect calls/jumps for objtool
    - x86/speculation, objtool: Annotate indirect calls/jumps for objtool on 32bit
    - x86/paravirt, objtool: Annotate indirect calls
    - x86/asm: Stop depending on ptrace.h in alternative.h
    - [Packaging] retpoline -- add safe usage hint support
    - [Packaging] retpoline-check -- only report additions
    - [Packaging] retpoline -- widen indirect call/jmp detection
    - [Packaging] retpoline -- elide %rip relative indirections
    - [Packaging] retpoline -- clear hint information from packages
    - SAUCE: modpost: add discard to non-allocatable whitelist
    - KVM: x86: Make indirect calls in emulator speculation safe
    - KVM: VMX: Make indirect call speculation safe
    - x86/boot, objtool: Annotate indirect jump in secondary_startup_64()
    - SAUCE: early/late -- annotate indirect calls in early/late initialisation
      code
    - SAUCE: vga_set_mode -- avoid jump tables
    - [Config] retpoline -- switch to new format
    - [Packaging] final-checks -- remove check for empty retpoline files

  * Xenial update to 4.4.117 stable release (LP: #1756860)
    - IB/mlx4: Fix incorrectly releasing steerable UD QPs when have only ETH ports
    - PM / devfreq: Propagate error from devfreq_add_device()
    - s390: fix handling of -1 in set{,fs}[gu]id16 syscalls
    - ARM: dts: STi: Add gpio polarity for "hdmi,hpd-gpio" property
    - arm: spear600: Add missing interrupt-parent of rtc
    - arm: spear13xx: Fix dmas cells
    - arm: spear13xx: Fix spics gpio controller's warning
    - ALSA: seq: Fix regression by incorrect ioctl_mutex usages
    - KVM/x86: Reduce retpoline performance impact in slot_handle_level_range(),
      by always inlining iterator helper methods
    - x86/cpu: Change type of x86_cache_size variable to unsigned int
    - drm/radeon: adjust tested variable
    - rtc-opal: Fix handling of firmware error codes, prevent busy loops
    - ext4: save error to disk in __ext4_grp_locked_error()
    - ext4: correct documentation for grpid mount option
    - mm: hide a #warning fo...

This bug was fixed in the package linux - 4.13.0-39.44

---------------
linux (4.13.0-39.44) artful; urgency=medium

  * linux: 4.13.0-39.44 -proposed tracker (LP: #1761456)

  * intel-microcode 3.20180312.0 causes lockup at login screen(w/ linux-
    image-4.13.0-37-generic) (LP: #1759920) // CVE-2017-5715 (Spectre v2
    Intel) // CVE-2017-5754
    - x86/mm: Reinitialize TLB state on hotplug and resume

  * intel-microcode 3.20180312.0 causes lockup at login screen(w/ linux-
    image-4.13.0-37-generic) (LP: #1759920) // CVE-2017-5715 (Spectre v2 Intel)
    - Revert "x86/mm: Only set IBPB when the new thread cannot ptrace current
      thread"
    - x86/speculation: Use Indirect Branch Prediction Barrier in context switch

  * DKMS driver builds fail with: Cannot use CONFIG_STACK_VALIDATION=y, please
    install libelf-dev, libelf-devel or elfutils-libelf-devel (LP: #1760876)
    - [Packaging] include the retpoline extractor in the headers

  * retpoline hints: primary infrastructure and initial hints (LP: #1758856)
    - [Packaging] retpoline-extract: flag *0xNNN(%reg) branches
    - x86/speculation, objtool: Annotate indirect calls/jumps for objtool
    - x86/speculation, objtool: Annotate indirect calls/jumps for objtool on 32bit
    - x86/paravirt, objtool: Annotate indirect calls
    - [Packaging] retpoline -- add safe usage hint support
    - [Packaging] retpoline-check -- only report additions
    - [Packaging] retpoline -- widen indirect call/jmp detection
    - [Packaging] retpoline -- elide %rip relative indirections
    - [Packaging] retpoline -- clear hint information from packages
    - KVM: x86: Make indirect calls in emulator speculation safe
    - KVM: VMX: Make indirect call speculation safe
    - x86/boot, objtool: Annotate indirect jump in secondary_startup_64()
    - SAUCE: early/late -- annotate indirect calls in early/late initialisation
      code
    - SAUCE: vga_set_mode -- avoid jump tables
    - [Config] retpoline -- switch to new format
    - [Packaging] retpoline hints -- handle missing files when RETPOLINE not
      enabled
    - [Packaging] final-checks -- remove check for empty retpoline files

  * retpoline: ignore %cs:0xNNN constant indirections (LP: #1752655)
    - [Packaging] retpoline -- elide %cs:0xNNNN constants on i386

  * zfs system process hung on container stop/delete (LP: #1754584)
    - SAUCE: Fix non-prefaulted page deadlock (LP: #1754584)

  * zfs-linux 0.6.5.11-1ubuntu5 ADT test failure with linux 4.15.0-1.2
    (LP: #1737761)
    - SAUCE: (noup) Update zfs to 0.6.5.11-1ubuntu3.2

  * AT_BASE_PLATFORM in AUXV is absent on kernels available on Ubuntu 17.10
    (LP: #1759312)
    - powerpc/64s: Fix NULL AT_BASE_PLATFORM when using DT CPU features

  * btrfs and tar sparse truncate archives (LP: #1757565)
    - Btrfs: move definition of the function btrfs_find_new_delalloc_bytes
    - Btrfs: fix reported number of inode blocks after buffered append writes

  * efifb broken on ThunderX-based Gigabyte nodes (LP: #1758375)
    - drivers/fbdev/efifb: Allow BAR to be moved instead of claiming it

  * Intel i40e PF reset due to incorrect MDD detection (continues...)
    (LP: #1723127)
    - i40e/i40ev...

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug was erroneously marked for verification in bionic; verification is not required and verification-needed-bionic is being removed.