zfs-2.1.4+ sru

Status changed to 'Confirmed' because the bug affects multiple users.

Regression & zsys testing passed out of bileto PPA, submitting for SRU.

Hello Dimitri, or anyone else affected,

Accepted zfs-linux into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zfs-linux/2.1.4-0ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

This bug was fixed in the package zfs-linux - 2.1.4-0ubuntu0.1

---------------
zfs-linux (2.1.4-0ubuntu0.1) jammy; urgency=medium

  * New upstream point release (LP: #1969482):
    - bufixes
    - performance fixes
    - v5.17 & v5.18 compat
    - scrub services
    - fix fallocate LP: #1969247

 -- Dimitri John Ledkov <email address hidden> Tue, 19 Apr 2022 15:44:23 +0100

I've set up proposed and installed following. It does NOT work as expected.

I used following command to install zfs / kernel packages and dependencies and it resulted in installation / upgrade of packages mentioned below.

apt install zfsutils-linux zfs-initramfs zfs-zed linux-generic linux-image-generic-hwe-22.04 linux-firmware linux-generic-hwe-20.04

Install:
linux-image-5.15.0-33-generic:amd64 (5.15.0-33.34, automatic),
linux-headers-5.15.0-33-generic:amd64 (5.15.0-33.34, automatic),
linux-modules-5.15.0-33-generic:amd64 (5.15.0-33.34, automatic),
linux-headers-5.15.0-33:amd64 (5.15.0-33.34, automatic),
linux-modules-extra-5.15.0-33-generic:amd64 (5.15.0-33.34, automatic)

Upgrade:
libnvpair3linux:amd64 (2.1.2-1ubuntu3, 2.1.4-0ubuntu0.1)
libuutil3linux:amd64 (2.1.2-1ubuntu3, 2.1.4-0ubuntu0.1),
libzpool5linux:amd64 (2.1.2-1ubuntu3, 2.1.4-0ubuntu0.1),
libzfs4linux:amd64 (2.1.2-1ubuntu3, 2.1.4-0ubuntu0.1),
zfsutils-linux:amd64 (2.1.2-1ubuntu3, 2.1.4-0ubuntu0.1)
zfs-zed:amd64 (2.1.2-1ubuntu3, 2.1.4-0ubuntu0.1),
zfs-initramfs:amd64 (2.1.2-1ubuntu3, 2.1.4-0ubuntu0.1)
linux-headers-generic:amd64 (5.15.0.30.33, 5.15.0.33.36),
linux-generic:amd64 (5.15.0.30.33, 5.15.0.33.36),
linux-image-generic:amd64 (5.15.0.30.33, 5.15.0.33.36)
linux-image-generic-hwe-22.04:amd64 (5.15.0.30.33, 5.15.0.33.36),
linux-headers-generic-hwe-22.04:amd64 (5.15.0.30.33, 5.15.0.33.36),
linux-generic-hwe-22.04:amd64 (5.15.0.30.33, 5.15.0.33.36)
linux-generic-hwe-20.04:amd64 (5.15.0.30.33, 5.15.0.33.36),
linux-firmware:amd64 (20220329.git681281e4-0ubuntu3, 20220329.git681281e4-0ubuntu3.2)

Then I got following system after reboot

root@XXX:~# uname -a
Linux nas139 5.15.0-33-generic #34-Ubuntu SMP Wed May 18 13:34:26 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

root@XXX:~# zfs --version
zfs-2.1.4-0ubuntu0.1
zfs-kmod-2.1.2-1ubuntu3

I still have old zfs-kmod a still very low ZFS performance on encrypted filesystem.
Did I do something wrong or there is missing updated zfs-kmod?

Hi, the verification you have performed is incorrect. You installed kernel and using zfs.ko as built into the kernel, which vendors zfs.ko from release/updates pockets only.

To test, zfs.ko from proposed, one has to install the dkms package, build it, enroll MOK (if secureboot is on, and MOK key is not already enrolled), and reboot.

I've installed zfs-dkms and it did the work. I can confirm system works as expected and performance is higher now.

# uname -a
Linux nas139 5.15.0-33-generic #34-Ubuntu SMP Wed May 18 13:34:26 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

# zfs --version
zfs-2.1.4-0ubuntu0.1
zfs-kmod-2.1.4-0ubuntu0.1

I'm setting this back to verification-needed as we still need to hear back from the kernel team about this part of the test plan:

 * kernel regression zfs testsuite pass

 * zsys integration test pass

I tested this version and it does fix a (severe) performance regression as mentioned here (#1977699) https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1977699. Would this be released before 21.10 reaches EOL?

I can also confirm that the fixes in -proposed fixes some severe performance issues on my system, with an encrypted ZFS root filesystem.

It should be noted that if you're using ZFS on an Intel machine with < 4 physical cores, these performance regressions are severe enough to cause the entire machine to noticeably stutter and freeze when the disk is accessed - so this is a fairly serious performance regression for a lot of laptop users!

I think the reason why it's so severe on this configuration is due to ZFS read/write spawning 4 tasks, which have a niceness of -20 causing the userspace to be starved (and perhaps security mitigations stopping/hindering the userspace from using hyperthreading).

kernel regression zfs testsuite has passed with v5.15.0-40-generic kernel and 2.1.4-0ubuntu0.1 dkms modules installed with stock zfs modules removed from the system. smoke, xfs-generic, fstest, stress all completed without any errors.

remaining testing is zsys installations and snapshots.

unencrypted zsys install worked, snapshotted, and booted into snapshot with v5.15.0-40-generic kernel and 2.1.4-0ubuntu0.1 dkms module.

encrypted zsys install worked, snapshotted, and booted into snapshot with v5.15.0-40-generic kernel and 2.1.4-0ubuntu0.1 dkms module.

Is there an ETA for releasing the fix to Jammy?

This bug was fixed in the package zfs-linux - 2.1.4-0ubuntu0.1

---------------
zfs-linux (2.1.4-0ubuntu0.1) jammy; urgency=medium

  * New upstream point release (LP: #1969482):
    - bufixes
    - performance fixes
    - v5.17 & v5.18 compat
    - scrub services
    - fix fallocate LP: #1969247

 -- Dimitri John Ledkov <email address hidden> Tue, 19 Apr 2022 15:44:23 +0100

The verification of the Stable Release Update for zfs-linux has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

I need an advice please:

Does this mean we need to switch to zfs-dkms instead of stock zfs to get the fix? Is this the long-term recommended solution?

As it stands today, with stock zfs, the zfs-kmod version is 2.1.2 hence contains the performance regression. Example on an up-to-date Jammy:

# zfs --version
zfs-2.1.4-0ubuntu0.1
zfs-kmod-2.1.2-1ubuntu3

@pdan

Ubuntu kernels vendor and build zfs.ko from the updates pocket. Meaning the next sru cycle of Ubuntu kernels will have the updated zfs.

If you want to receive zfs.ko ahead of time, you can install zfs-dkms for a little bit, until it lands in the kernel builds. Or just wait a little bit longer.

@dmitri

Thank you! Quick question, the Jammy kernel version was just bumped, but it still does not include 2.1.4 zfs module

# uname -a
Linux arpa 5.15.0-41-generic #44-Ubuntu SMP Wed Jun 22 14:20:53 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

# zfs --version
zfs-2.1.4-0ubuntu0.1
zfs-kmod-2.1.2-1ubuntu3

yes..... because that's the old sru cycle. the kernel was already built on 22nd of June and was undergoing validation. This zfs update was released to updates after that, on 28th of June.

the next sru cycle that will pick up updated zfs has not started yet.

Got it, thank you. It sounds like testing encrypted zfs performance could perhaps be added to the validation suite for kernels.

The updated kernel was released, and it fixes the performance regression. Thank you!

# uname -a
Linux arpa 5.15.0-43-generic #46-Ubuntu SMP Tue Jul 12 10:30:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

# zfs --version
zfs-2.1.4-0ubuntu0.1
zfs-kmod-2.1.4-0ubuntu0.1

# grep . /sys/module/icp/parameters/*impl*
/sys/module/icp/parameters/icp_aes_impl:cycle [fastest] generic x86_64 aesni
/sys/module/icp/parameters/icp_gcm_impl:cycle [fastest] avx generic pclmulqdq

# dd if=14GBfile.tmp of=/dev/null bs=1M
13411+1 records in
13411+1 records out
14062902185 bytes (14 GB, 13 GiB) copied, 12.6139 s, 1.1 GB/s