LXD containers using shiftfs on ZFS or TMPFS broken on 5.15.0-48.54

Added sudo systemctl reload snap.lxd.daemon after disabling shiftfs.

This change was made by a bot.

Suspect is:

  * refactoring of overlayfs fix to properly support shiftfs (LP: #1983640)
    - SAUCE: overlayfs: remove CONFIG_AUFS_FS dependency

I like the reproducer, i wonder if i should add it to zfs-linux as a test case. Or maybe to the kernel.

(zfs-linux portion is to autopkgtest the reproducer, because this has been pain multiple times now)

This bug also breaks `lxc file push` functionality.

Anbox Cloud is affected by this too. We used shiftfs by default until our 1.14 release in May 2022. According to the metrics we have there are still a few users around with the 1.13 release which enables shiftfs on ZFS.

Status changed to 'Confirmed' because the bug affects multiple users.

Is there any plan or estimation on the fix?

Dear friends,

Right now I'm on the way to understand what's happening with this.

# strace touch b
execve("/usr/bin/touch", ["touch", "b"], 0x7ffd29f848a8 /* 7 vars */) = 0
brk(NULL) = 0x56007dba6000
arch_prctl(0x3001 /* ARCH_??? */, 0x7fff436afcb0) = -1 EINVAL (Invalid argument)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6bb5d2d000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
<...>
openat(AT_FDCWD, "b", O_WRONLY|O_CREAT|O_NOCTTY|O_NONBLOCK, 0666) = -1 EOVERFLOW (Value too large for defined data type)

From ftrace/perf trace:
  877.582556 | 0) | /* do_sys_openat2__return: (__x64_sys_openat+0x55/0x90 <- do_sys_openat2) arg1=0xffffffffffffffb5 */

0xffffffffffffffb5 it's 2's complement of -EOVERFLOW error

I've finally traced it to:
static inline int may_create(struct user_namespace *mnt_userns,
        struct inode *dir, struct dentry *child)
{
 audit_inode_child(dir, child, AUDIT_TYPE_CHILD_CREATE);
 if (child->d_inode)
  return -EEXIST;
 if (IS_DEADDIR(dir))
  return -ENOENT;
 if (!fsuidgid_has_mapping(dir->i_sb, mnt_userns))
  return -EOVERFLOW; // <<< looks like error comes from here

My suspicion is that the problem is caused by these two changes, which are potentially incompatible with shiftfs:
+ - fs: tweak fsuidgid_has_mapping()
+ - fs: support mapped mounts of mapped filesystems
(changelog from https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/commit/?h=Ubuntu-5.15.0-48.54&id=941bdeb5ab2258758fce5f4d06296da98bfa7e82)

Will continue investigation.

This degradation is not related to
SAUCE: overlayfs: remove CONFIG_AUFS_FS dependency
change

Draft patch is ready
https://lists.ubuntu.com/archives/kernel-team/2022-November/134883.html

The attachment "0001-UBUNTU-SAUCE-shiftfs-fix-permanent-EOVERFLOW-inside-.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

Status changed to 'Confirmed' because the bug affects multiple users.

Any updates?

I've tested the patch on kernel 5.15.0-57.63 and it solves the issue for me.

Aleksandr - Andrea asked you in your kernel-team thread to rebase the patch on master-next, hopefully if you can do this it'll be accepted. Keen to see this issue fixed in jammy as it's breaking Docker+Sysbox containers too.

We are waiting for the maintainer's decision on that patch. I've done everything from my side. I'm sorry that it takes too long to be landed in the kernel.

https://lists.ubuntu.com/archives/kernel-team/2023-January/136719.html

https://lists.ubuntu.com/archives/kernel-team/2023-January/136731.html

This bug is awaiting verification that the linux/5.19.0-35.36 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-kinetic' to 'verification-done-kinetic'. If the problem still exists, change the tag 'verification-needed-kinetic' to 'verification-failed-kinetic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug was fixed in the package linux - 5.19.0-35.36

---------------
linux (5.19.0-35.36) kinetic; urgency=medium

  * kinetic/linux: 5.19.0-35.36 -proposed tracker (LP: #2004652)

  * CVE-2023-0461
    - SAUCE: Fix inet_csk_listen_start after CVE-2023-0461

linux (5.19.0-34.35) kinetic; urgency=medium

  * kinetic/linux: 5.19.0-34.35 -proposed tracker (LP: #2004299)

  * LXD containers using shiftfs on ZFS or TMPFS broken on 5.15.0-48.54
    (LP: #1990849)
    - [SAUCE] shiftfs: fix -EOVERFLOW inside the container

  * Kinetic update: upstream stable patchset 2023-01-27 (LP: #2004051)
    - ASoC: fsl_sai: use local device pointer
    - serial: Add rs485_supported to uart_port
    - serial: fsl_lpuart: Fill in rs485_supported
    - x86/sgx: Create utility to validate user provided offset and length
    - x86/sgx: Add overflow check in sgx_validate_offset_length()
    - binder: validate alloc->mm in ->mmap() handler
    - ceph: Use kcalloc for allocating multiple elements
    - ceph: fix NULL pointer dereference for req->r_session
    - wifi: mac80211: fix memory free error when registering wiphy fail
    - wifi: mac80211_hwsim: fix debugfs attribute ps with rc table support
    - riscv: dts: sifive unleashed: Add PWM controlled LEDs
    - audit: fix undefined behavior in bit shift for AUDIT_BIT
    - wifi: airo: do not assign -1 to unsigned char
    - wifi: mac80211: Fix ack frame idr leak when mesh has no route
    - wifi: ath11k: Fix QCN9074 firmware boot on x86
    - spi: stm32: fix stm32_spi_prepare_mbr() that halves spi clk for every run
    - selftests/bpf: Add verifier test for release_reference()
    - Revert "net: macsec: report real_dev features when HW offloading is enabled"
    - platform/x86: ideapad-laptop: Disable touchpad_switch
    - platform/x86: touchscreen_dmi: Add info for the RCA Cambio W101 v2 2-in-1
    - platform/x86/intel/pmt: Sapphire Rapids PMT errata fix
    - scsi: ibmvfc: Avoid path failures during live migration
    - scsi: scsi_debug: Make the READ CAPACITY response compliant with ZBC
    - drm: panel-orientation-quirks: Add quirk for Acer Switch V 10 (SW5-017)
    - block, bfq: fix null pointer dereference in bfq_bio_bfqg()
    - arm64/syscall: Include asm/ptrace.h in syscall_wrapper header.
    - nvmet: fix memory leak in nvmet_subsys_attr_model_store_locked
    - Revert "drm/amdgpu: Revert "drm/amdgpu: getting fan speed pwm for vega10
      properly""
    - ALSA: usb-audio: add quirk to fix Hamedal C20 disconnect issue
    - RISC-V: vdso: Do not add missing symbols to version section in linker script
    - MIPS: pic32: treat port as signed integer
    - xfrm: fix "disable_policy" on ipv4 early demux
    - xfrm: replay: Fix ESN wrap around for GSO
    - af_key: Fix send_acquire race with pfkey_register
    - ARM: dts: am335x-pcm-953: Define fixed regulators in root node
    - ASoC: hdac_hda: fix hda pcm buffer overflow issue
    - ASoC: sgtl5000: Reset the CHIP_CLK_CTRL reg on remove
    - ASoC: soc-pcm: Don't zero TDM masks in __soc_pcm_open()
    - x86/hyperv: Restore VP assist page after cpu offlining/onlining
    - scsi: storvsc: Fix handling of srb_status and capacity change events
    - ASoC: max983...

This bug is awaiting verification that the linux/5.15.0-68.75 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-aws/5.19.0-1021.22 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-kinetic' to 'verification-done-kinetic'. If the problem still exists, change the tag 'verification-needed-kinetic' to 'verification-failed-kinetic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-azure/5.19.0-1022.23 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-kinetic' to 'verification-done-kinetic'. If the problem still exists, change the tag 'verification-needed-kinetic' to 'verification-failed-kinetic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-starfive/5.19.0-1014.16 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-kinetic' to 'verification-done-kinetic'. If the problem still exists, change the tag 'verification-needed-kinetic' to 'verification-failed-kinetic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Please, forgive the newbie question.

I had to manually install the 5.15-0-47-generic kernel on 22.04.2 LTS to work around the ZFS shiftfs problem as it impacted production LXD containers which relied upon local ZFS storage access within the containers.

Does the kernel fix resolve the issue for this scenario or does zfs-linux (ubuntu) also require a patch?

When does 'fix released' reach production? (help info says at next release) Is that a Ubuntu Server version release or update release?

How do I verify that a production Ubuntu Server release / update will contain the bug fix?

Where is the patch list for a release / update published so I can verify this fix is included?

@Matthew, that's a good question. Since you are using 22.04 with the 5.15 kernel, you'll need to wait for the 5.15.0-68.75 kernel to land in -updates as it is still in -proposed ATM.

The zfs-linux user space tools, shouldn't need an update for this bug but even if it did, the LXD snap bundles recent versions of it so you are covered.

The changes that went into the -68.75 kernel are listed here: https://changelogs.ubuntu.com/changelogs/pool/main/l/linux/linux_5.15.0-68.75/changelog. If you are subscribed to this bug, you should be notified when the fix actually lands in -updates.

This bug is awaiting verification that the linux-riscv/5.19.0-1015.16 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-kinetic' to 'verification-done-kinetic'. If the problem still exists, change the tag 'verification-needed-kinetic' to 'verification-failed-kinetic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-allwinner/5.19.0-1009.9 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-kinetic' to 'verification-done-kinetic'. If the problem still exists, change the tag 'verification-needed-kinetic' to 'verification-failed-kinetic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug was fixed in the package linux - 6.2.0-18.18

---------------
linux (6.2.0-18.18) lunar; urgency=medium

  * lunar/linux: 6.2.0-18.18 -proposed tracker (LP: #2011750)

  * lunar/linux 6.2 fails to boot on arm64 (LP: #2011748)
    - SAUCE: Revert "efi: random: fix NULL-deref when refreshing seed"
    - SAUCE: Revert "efi: random: refresh non-volatile random seed when RNG is
      initialized"

 -- Andrea Righi <email address hidden> Wed, 15 Mar 2023 23:54:18 +0100

This bug was fixed in the package linux - 5.15.0-69.76

---------------
linux (5.15.0-69.76) jammy; urgency=medium

  * jammy/linux: 5.15.0-69.76 -proposed tracker (LP: #2012092)

  * NFS deathlock with last Kernel 5.4.0-144.161 and 5.15.0-67.74 (LP: #2009325)
    - NFS: Correct timing for assigning access cache timestamp

linux (5.15.0-68.75) jammy; urgency=medium

  * jammy/linux: 5.15.0-68.75 -proposed tracker (LP: #2008349)

  * Packaging resync (LP: #1786013)
    - debian/dkms-versions -- update from kernel-versions (main/2023.02.27)

  * Ubuntu 22.04 kernel 5.15.0-46-generic leaks kernel memory in kmalloc-2k
    slabs (LP: #1987430)
    - SAUCE: audit: fix memory leak of audit_log_lsm()

  * [EGS] Backport intel_idle support for Eagle Stream Ubuntu 22.04 release
    (LP: #2003267)
    - intel_idle: add SPR support
    - intel_idle: add 'preferred_cstates' module argument
    - intel_idle: add core C6 optimization for SPR
    - cpuidle: intel_idle: Drop redundant backslash at line end
    - intel_idle: Fix the 'preferred_cstates' module parameter
    - intel_idle: Fix SPR C6 optimization
    - intel_idle: make SPR C1 and C1E be independent

  * Fix speaker mute hotkey doesn't work on Dell G16 series (LP: #2003161)
    - platform/x86: dell-wmi: Add a keymap for KEY_MUTE in type 0x0010 table

  * Fix the ACPI _CPC not found error from kernel dmesg on some dynamic SSDT
    table loaded firmwares (LP: #2006077)
    - ACPI: bus: Avoid using CPPC if not supported by firmware
    - ACPI: bus: Set CPPC _OSC bits for all and when CPPC_LIB is supported
    - ACPI: CPPC: Only probe for _CPC if CPPC v2 is acked

  * rtcpie in timers from ubuntu_kernel_selftests randomly failing
    (LP: #1814234)
    - SAUCE: selftest: rtcpie: Force passing unreliable subtest

  * Jammy update: v5.15.87 upstream stable release (LP: #2007441)
    - usb: dwc3: qcom: Fix memory leak in dwc3_qcom_interconnect_init
    - cifs: fix oops during encryption
    - nvme-pci: fix doorbell buffer value endianness
    - nvme-pci: fix mempool alloc size
    - nvme-pci: fix page size checks
    - ACPI: resource: do IRQ override on LENOVO IdeaPad
    - ACPI: resource: do IRQ override on XMG Core 15
    - ACPI: resource: do IRQ override on Lenovo 14ALC7
    - block, bfq: fix uaf for bfqq in bfq_exit_icq_bfqq
    - ata: ahci: Fix PCS quirk application for suspend
    - nvme: fix the NVME_CMD_EFFECTS_CSE_MASK definition
    - nvmet: don't defer passthrough commands with trivial effects to the
      workqueue
    - fs/ntfs3: Validate BOOT record_size
    - fs/ntfs3: Add overflow check for attribute size
    - fs/ntfs3: Validate data run offset
    - fs/ntfs3: Add null pointer check to attr_load_runs_vcn
    - fs/ntfs3: Fix memory leak on ntfs_fill_super() error path
    - fs/ntfs3: Add null pointer check for inode operations
    - fs/ntfs3: Validate attribute name offset
    - fs/ntfs3: Validate buffer length while parsing index
    - fs/ntfs3: Validate resident attribute name
    - fs/ntfs3: Fix slab-out-of-bounds read in run_unpack
    - soundwire: dmi-quirks: add quirk variant for LAPBC710 NUC15
    - fs/ntfs3: Validate index root when initialize NTFS security
    - fs/ntfs3: Use __G...

This bug is awaiting verification that the linux-azure/5.15.0-1036.43 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-aws/5.15.0-1034.38 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-nvidia-5.19/5.19.0-1009.9 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-xilinx-zynqmp/5.15.0-1021.25 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Ubuntu 22.10 (Kinetic Kudu) has reached end of life, so this bug will not be fixed for that specific release.

This bug is awaiting verification that the linux-aws-5.15/5.15.0-1046.51~20.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal-linux-aws-5.15' to 'verification-done-focal-linux-aws-5.15'. If the problem still exists, change the tag 'verification-needed-focal-linux-aws-5.15' to 'verification-failed-focal-linux-aws-5.15'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!