EFI pstore not cleared on boot

Bug #1978079 reported by Alexander Graf
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux-bluefield (Ubuntu)
New
Undecided
William Tu
Focal
Fix Released
Undecided
William Tu
Jammy
Fix Committed
Undecided
William Tu
systemd (Ubuntu)
Fix Released
Undecided
Nick Rosbrook
Focal
Fix Released
Undecided
Mustafa Kemal Gilor
Impish
Won't Fix
Undecided
Unassigned
Jammy
Fix Released
Undecided
Mustafa Kemal Gilor
Kinetic
Fix Released
Undecided
Nick Rosbrook

Bug Description

[Impact]

Systemd has a systemd-pstore component that scans the pstore on boot and if non-empty, takes all previously created dumps, transfers them into its journal and removes the pstore elements. This is very important on UEFI systems, which only have a limited amount of space for variables.

In Ubuntu, the kernel is configured with CONFIG_EFI_VARS_PSTORE=m which means the EFI pstore support gets loaded dynamically. In all of my boots, this dynamic module loading happened *after* systemd tried to check for pstore variables. So systemd-pstore never starts and never clears the UEFI variable store. I see this happening in AWS on Graviton instances, which eventually run out of space to store the dumps. On real hardware, this behavior may lead to unbootable systems.

```
$ systemctl status systemd-pstore
○ systemd-pstore.service - Platform Persistent Storage Archival
     Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
  Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago
             └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met
       Docs: man:systemd-pstore(8)

Jun 09 09:11:41 ip-172-31-0-61 systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped.

$ ls -la /sys/fs/pstore
total 0
drwxr-x--- 2 root root 0 Jun 9 09:11 .
drwxr-xr-x 8 root root 0 Jun 9 09:11 ..
-r--r--r-- 1 root root 1803 Jun 9 09:07 dmesg-efi-165476562001001
-r--r--r-- 1 root root 1777 Jun 9 09:07 dmesg-efi-165476562002001
-r--r--r-- 1 root root 1773 Jun 9 09:07 dmesg-efi-165476562003001
-r--r--r-- 1 root root 1815 Jun 9 09:07 dmesg-efi-165476562004001
-r--r--r-- 1 root root 1826 Jun 9 09:07 dmesg-efi-165476562005001
-r--r--r-- 1 root root 1754 Jun 9 09:07 dmesg-efi-165476562006001
-r--r--r-- 1 root root 1821 Jun 9 09:07 dmesg-efi-165476562007001
-r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562008001
-r--r--r-- 1 root root 1729 Jun 9 09:07 dmesg-efi-165476562009001
-r--r--r-- 1 root root 1819 Jun 9 09:07 dmesg-efi-165476562010001
-r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562011001
-r--r--r-- 1 root root 1775 Jun 9 09:07 dmesg-efi-165476562012001
-r--r--r-- 1 root root 1802 Jun 9 09:07 dmesg-efi-165476562013001
-r--r--r-- 1 root root 1812 Jun 9 09:07 dmesg-efi-165476562014001
-r--r--r-- 1 root root 1764 Jun 9 09:07 dmesg-efi-165476562015001
-r--r--r-- 1 root root 1795 Jun 9 09:11 dmesg-efi-165476589801001
-r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589802001
-r--r--r-- 1 root root 1683 Jun 9 09:11 dmesg-efi-165476589803001
-r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589804001
-r--r--r-- 1 root root 1771 Jun 9 09:11 dmesg-efi-165476589805001
-r--r--r-- 1 root root 1797 Jun 9 09:11 dmesg-efi-165476589806001
-r--r--r-- 1 root root 1805 Jun 9 09:11 dmesg-efi-165476589807001
-r--r--r-- 1 root root 1781 Jun 9 09:11 dmesg-efi-165476589808001
-r--r--r-- 1 root root 1806 Jun 9 09:11 dmesg-efi-165476589809001
-r--r--r-- 1 root root 1821 Jun 9 09:11 dmesg-efi-165476589810001
-r--r--r-- 1 root root 1763 Jun 9 09:11 dmesg-efi-165476589811001
-r--r--r-- 1 root root 1783 Jun 9 09:11 dmesg-efi-165476589812001
-r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589813001
-r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589814001
-r--r--r-- 1 root root 1786 Jun 9 09:11 dmesg-efi-165476589815001
```

This problem affects (at least) Ubuntu 20.04 and 22.04. A quick fix would be to configure CONFIG_EFI_VARS_PSTORE=y so that it's always available. A long term fix would make systemd rescan the directory after all module probing settled.

[Test Plan]

In order to be able to reproduce this issue, the system must have EFI-backed pstore.

To check which kind of backend that pstore, use `cat /sys/module/pstore/parameters/backend`

If it says `efi`, the steps below are applicable. Otherwise, find an environment that has EFI backed pstore.

# Enable the pstore service. This service is supposed to move the data in /sys/fs/pstore
# to the `/var/lib/systemd/pstore` path on boot.
systemctl enable systemd-pstore.service # (or can be vendor enabled)

# Crash the kernel
echo 1 > /proc/sys/kernel/sysrq
echo 1 > /proc/sys/kernel/panic # this is usually set to zero, causing kernel to loop over the panic and freeze
echo "c" > /proc/sysrq-trigger

# The system will reboot itself. Check `/sys/fs/pstore` path first:
ls /sys/fs/pstore # The path should not be empty, which means the systemd-pstore has failed to do its' job
ls /var/lib/systemd/pstore # The path should be empty.

# Apply the fix
sudo add-apt-repository ppa:mustafakemalgilor/lp-1978079-1
sudo apt upgrade

# Crash the kernel
echo 1 > /proc/sys/kernel/sysrq
echo 1 > /proc/sys/kernel/panic # this is usually set to zero, causing kernel to loop over the panic and freeze
echo "c" > /proc/sysrq-trigger

# The system will reboot itself. After reboot, the contents of the `/sys/fs/pstore` must have been moved to the `/var/lib/systemd/pstore` path.
ls /sys/fs/pstore # The path should be empty
ls /var/lib/systemd/pstore # The path should not be empty

[Where problems could occur]

On some systems, even though the described bug is present, the effect of the bug could not be observed. The nature of the issue suggests that this is a due to a timing issue.

Related branches

Revision history for this message
Alexander Graf (grafaws) wrote :

This systemd issue describes the problem above as well: https://github.com/systemd/systemd/issues/18540

Revision history for this message
Alexander Graf (grafaws) wrote :
Revision history for this message
Alexander Graf (grafaws) wrote :

The upstream pull request has been accepted, please backport the patch to (at least) Ubuntu 20.04 and 22.04.

Nick Rosbrook (enr0n)
tags: added: rls-ff-incoming rls-jj-incoming
Changed in systemd (Ubuntu):
assignee: nobody → Mustafa Kemal Gilor (mustafakemalgilor)
status: New → In Progress
tags: added: fr-2484
tags: removed: rls-ff-incoming rls-jj-incoming
tags: added: seg sts sts-sponsor
tags: added: ubuntu-sponsors
description: updated
Revision history for this message
Nick Rosbrook (enr0n) wrote :

For the kinetic portion of this fix, I can pull the patch in while I am finishing the v251 merge this month.

Revision history for this message
Mustafa Kemal Gilor (mustafakemalgilor) wrote :

Sounds good, deleting the merge proposal for the kinetic. Corrected the target repository for the focal & jammy MP's.

tags: added: sts-sponsor-dgadomski
Lukas Märdian (slyon)
Changed in systemd (Ubuntu Jammy):
status: New → In Progress
Changed in systemd (Ubuntu Focal):
status: New → In Progress
assignee: nobody → Mustafa Kemal Gilor (mustafakemalgilor)
Changed in systemd (Ubuntu Jammy):
assignee: nobody → Mustafa Kemal Gilor (mustafakemalgilor)
Changed in systemd (Ubuntu Kinetic):
assignee: Mustafa Kemal Gilor (mustafakemalgilor) → Nick Rosbrook (enr0n)
Revision history for this message
Lukas Märdian (slyon) wrote (last edit ):

Thank you very much!
This resembles the upstream fix and LGTM.

Good work on the detailed reproducer/testcase and thanks for executing autopkgtest ahead of time!
Keep up the good work!

I've staged both proposals in the git tree, to be landed with the next systemd SRUs:
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/log/?h=ubuntu-jammy
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/log/?h=ubuntu-focal

Unfortunately the last SRUs just went out a few days ago for Jammy and a month ago for Focal, it might take a while for the next one to happen. But given that this issue has an easy workaround this shouldn't be a problem IMO.
The next systemd SRUs most probably won't go out before Impish EOL, so I'm marking this task "WONTFIX".

As a workaround the service unit dependency settings can easily be adopted, using a drop-in config:
$ cat /etc/systemd/system/systemd-pstore.service.d/override.conf
[Unit]
After=modprobe@efi_pstore.service <email address hidden> modprobe@chromeos_pstore.service <email address hidden> modprobe@pstore_zone.service modprobe@pstore_blk.service
Wants=modprobe@efi_pstore.service <email address hidden> modprobe@chromeos_pstore.service <email address hidden> modprobe@pstore_zone.service modprobe@pstore_blk.service

Changed in systemd (Ubuntu Impish):
status: New → Won't Fix
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Alexander, or anyone else affected,

Accepted systemd into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/249.11-0ubuntu3.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Jammy):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-jammy
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (systemd/249.11-0ubuntu3.4)

All autopkgtests for the newly accepted systemd (249.11-0ubuntu3.4) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

prometheus-libvirt-exporter/unknown (s390x)
corosync-qdevice/unknown (ppc64el, s390x)
csync2/unknown (ppc64el, s390x)
pyudev/unknown (ppc64el)
dlm/unknown (s390x)
systemd/unknown (ppc64el)
tpm2-abrmd/unknown (ppc64el)
dbus/unknown (ppc64el)
linux-lowlatency/5.15.0-40.43 (arm64)
samba/unknown (ppc64el)
dovecot/unknown (ppc64el, arm64)
systemd/249.11-0ubuntu3.4 (armhf)
prometheus-exporter-exporter/unknown (ppc64el)
conntrack-tools/unknown (ppc64el, s390x)
cups/unknown (s390x)
dpdk/unknown (ppc64el, arm64)
network-manager/1.36.6-0ubuntu2 (arm64)
php8.1/unknown (s390x)
procps/unknown (ppc64el)
gvfs/1.48.2-0ubuntu1 (ppc64el)
tgt/unknown (ppc64el)
redis/unknown (ppc64el)
qlcplus/unknown (ppc64el, s390x)
prometheus-squid-exporter/unknown (s390x)
flatpak/1.12.7-1 (amd64)
dq/unknown (arm64)
rtkit/unknown (ppc64el)
cockpit/unknown (s390x)
netplan.io/unknown (s390x)
pdns-recursor/unknown (s390x)
remctl/unknown (ppc64el)
comitup/unknown (s390x)
policykit-1/unknown (ppc64el)
rust-whoami/unknown (ppc64el)
casync/2+20201210-1build1 (ppc64el)
dbus-broker/unknown (ppc64el, s390x)
rpcbind/unknown (ppc64el)
openzwave/unknown (s390x)
netplan.io/0.104-0ubuntu2 (amd64, arm64)
postgresql-14/unknown (ppc64el)
python-uinput/unknown (s390x)
libsoup2.4/unknown (ppc64el)
libsfml/unknown (ppc64el)
polkit-qt-1/unknown (ppc64el)
debspawn/unknown (ppc64el)
corosync/unknown (ppc64el, s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Lukas Märdian (slyon)
Changed in systemd (Ubuntu Kinetic):
status: In Progress → Fix Committed
Revision history for this message
Lukas Märdian (slyon) wrote :

All autopkgtest regressions have been resolved by re-running them, using proper triggers.

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (22.3 KiB)

This bug was fixed in the package systemd - 251.2-2ubuntu1

---------------
systemd (251.2-2ubuntu1) kinetic; urgency=medium

  [ Nick Rosbrook ]
  * Merge to Ubuntu from Debian unstable
    - Dropped patches (applied upstream)
      + test-explicitly-configure-oomd-stuff-via-dropins.patch
      + test-enable-systemd-oomd.service.patch
      + linux-5.15-compat-ioprio/shared-split-out-ioprio-related-stuff-into-ioprio-util.-c.patch
      + linux-5.15-compat-ioprio/variuos-add-missing-includes.patch
      + linux-5.15-compat-ioprio/man-don-t-mention-IOSchedulingClass-none-anymore-in-the-d.patch
      + linux-5.15-compat-ioprio/test-add-test-for-ioprio-normalization.patch
      + linux-5.15-compat-ioprio/Define-ioprio_-get-set-the-same-as-other-compat-syscalls.patch
      + linux-5.15-compat-ioprio/Get-rid-of-ioprio.h-and-add-a-minimalistic-reimplementati.patch
      + linux-5.15-compat-ioprio/ioprio-util-add-macro-for-default-ioprio-settings.patch
      + linux-5.15-compat-ioprio/ioprio-normalize-io-priority-values-in-configuration.patch
      + linux-5.15-compat-ioprio/core-normalize-ioprio-values-we-acquire-from-kernel.patch
      + test-also-show-the-memory-pressure-of-testchill.service.patch
      + test-make-test-55-oomd-less-flaky.patch
      + lp1964494-network-do-not-enable-IPv4-ACD-for-IPv4-link-local-a.patch
      + lp1966381-oomd-calculate-used-memory-with-MemAvailable-instead-of-M.patch
      + lp1926860-hwdb-remove-the-tablet-pad-entry-for-the-UC-Logic-1060N.patch
      + oomd-move-oomctl-to-bindir.patch
      + test-enable-debug-logging-of-systemd-oomd.patch
      + lp1943561/Add-additional-Dell-models-that-require-ACCEL_LOCATION-ba.patch
      + lp1943561/Use-SKU-to-identify-Dell-clamshell-models-for-acceleromet.patch
      + lp1929345/hwdb-Force-release-calculator-key-on-all-HP-OMEN-laptops.patch
      + lp1929345/hwdb-Add-force-release-for-HP-Omen-15-calculator-key.-205.patch
      + sysusers-split-up-systemd.conf.patch
      + hwdb-Add-mic-mute-key-mapping-for-HP-Elite-x360.patch
      + test-check-memory-pressure-more-frequently.patch
      + meson-minor-cleanup.patch
      + units-don-t-install-dbus-org.freedesktop.oom1.service-ali.patch
      + lp1950508-cgroup-check-if-any-controller-is-in-use-as-v1.patch
      + lp1952735-keymap-Add-microphone-mute-keymap-for-Dell-Machine.patch
      + test-tweak-parameters-for-TEST-55-OOMD.patch
      + deny-list-TEST-29-PORTABLE-and-TEST-50-DISSECT.patch
      + lp1955997-unmask-intel-hid-for-HP-machines.patch
      + lp1952733-hwdb-60-keyboard-Update-Dell-Privacy-Micmute-Hotkey-Map.patch
      + Merge-pull-request-20705-from-yuwata-test-oomd-util.patch
    - Refreshed patches
      + debian/Ubuntu-UseDomains-by-default.patch
      + debian/UBUNTU-Support-system-image-read-only-etc.patch
      + Revert-network-if-sys-is-rw-then-udev-should-be-around.patch
      + debian/UBUNTU-src-test-testmount-util.c-Skip-parts-of-test-mount-util-in-LXC.patch
      + lp1950794-Revert-sd-dhcp-do-not-use-detect_container-to-guess-.patch
      + 0001-Revert-tests-add-test-case-for-UMask-BindPaths-combi.patch
  * Drop debian/Ubuntu-core-in-execute-soft-fail-setting-Nice-priority-when.patch.
    This patch...

Changed in systemd (Ubuntu Kinetic):
status: Fix Committed → Fix Released
Revision history for this message
Mustafa Kemal Gilor (mustafakemalgilor) wrote (last edit ):

Verification done for jammy:

- Environment -

root@host:/tmp# uname -a
Linux host 5.15.0-40-generic #43-Ubuntu SMP Wed Jun 15 12:54:21 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

root@host:/tmp# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

root@host:/tmp# systemd --version
systemd 249 (249.11-0ubuntu3.4)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP -LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

root@host:/tmp# cat /sys/module/pstore/parameters/backend
efi

-----------------

- Test steps -

See [Test Plan]

-----------------

- Result -

Verification : OK

root@host:/tmp# ls /sys/fs/pstore/ # empty
root@host:/tmp# ls /var/lib/systemd/pstore/ # non-empty
165700416

root@host:/tmp# systemctl status systemd-pstore.service
● systemd-pstore.service - Platform Persistent Storage Archival
     Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled)
     Active: active (exited) since Tue 2022-07-05 09:58:43 +03; 1min 8s ago
       Docs: man:systemd-pstore(8)
    Process: 1693 ExecStart=/lib/systemd/systemd-pstore (code=exited, status=0/SUCCESS)
   Main PID: 1693 (code=exited, status=0/SUCCESS)
        CPU: 6ms

Tem 05 09:58:43 host systemd-pstore[1693]: PStore dmesg-efi-165700416809001 moved to /var/lib/systemd/pstore/165700416/dmesg-efi-165700416809001
Tem 05 09:58:43 host systemd-pstore[1693]: PStore dmesg-efi-165700416808001 moved to /var/lib/systemd/pstore/165700416/dmesg-efi-165700416808001
Tem 05 09:58:43 host systemd-pstore[1693]: PStore dmesg-efi-165700416807001 moved to /var/lib/systemd/pstore/165700416/dmesg-efi-165700416807001
Tem 05 09:58:43 host systemd-pstore[1693]: PStore dmesg-efi-165700416806001 moved to /var/lib/systemd/pstore/165700416/dmesg-efi-165700416806001
Tem 05 09:58:43 host systemd-pstore[1693]: PStore dmesg-efi-165700416805001 moved to /var/lib/systemd/pstore/165700416/dmesg-efi-165700416805001
Tem 05 09:58:43 host systemd-pstore[1693]: PStore dmesg-efi-165700416804001 moved to /var/lib/systemd/pstore/165700416/dmesg-efi-165700416804001
Tem 05 09:58:43 host systemd-pstore[1693]: PStore dmesg-efi-165700416803001 moved to /var/lib/systemd/pstore/165700416/dmesg-efi-165700416803001
Tem 05 09:58:43 host systemd-pstore[1693]: PStore dmesg-efi-165700416802001 moved to /var/lib/systemd/pstore/165700416/dmesg-efi-165700416802001
Tem 05 09:58:43 host systemd-pstore[1693]: PStore dmesg-efi-165700416701001 moved to /var/lib/systemd/pstore/165700416/dmesg-efi-165700416701001
Tem 05 09:58:43 host systemd[1]: Finished Platform Persistent Storage Archival.

tags: added: verification-done-jammy
removed: verification-needed-jammy
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 249.11-0ubuntu3.4

---------------
systemd (249.11-0ubuntu3.4) jammy; urgency=medium

  [ Mustafa Kemal Gilor ]
  * d/p/lp1978079-efi-pstore-not-cleared-on-boot.patch: pstore: Run after
    modules are loaded. Thanks to Alexander Graf <email address hidden>.
    (LP: #1978079)
    Author: Mustafa Kemal Gilor
    File: debian/patches/lp1978079-efi-pstore-not-cleared-on-boot.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=d990b13612810a296246011ad66a165b30166702

  [ Nick Rosbrook ]
  * systemd-oomd: set ManagedOOMSwap=auto on -.slice (LP: #1972159)
    This has the effect of disabling swap kill by default, so cgroups will
    only be monitored for memory pressure, and not swap usage.
    File: debian/extra/systemd-oomd-defaults/-.slice.d/10-oomd-root-slice-defaults.conf
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=e93c944c58ec376454301e9c9b55d35be7c14a89

 -- Nick Rosbrook <email address hidden> Mon, 27 Jun 2022 14:28:46 -0400

Changed in systemd (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Alexander, or anyone else affected,

Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.18 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed-focal
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (systemd/245.4-4ubuntu3.18)

All autopkgtests for the newly accepted systemd (245.4-4ubuntu3.18) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

linux-gke-5.15/5.15.0-1015.18~20.04.1 (arm64, amd64)
netplan.io/0.104-0ubuntu2~20.04.2 (armhf)
linux-hwe-5.15/5.15.0-48.54~20.04.1 (armhf, amd64)
gvfs/1.44.1-1ubuntu1.1 (ppc64el)
linux-oracle-5.15/5.15.0-1016.20~20.04.1 (arm64)
linux-oracle-5.13/5.13.0-1036.43~20.04.1 (arm64)
linux-hwe-5.11/5.11.0-61.61 (arm64)
systemd/245.4-4ubuntu3.18 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Nick Rosbrook (enr0n) wrote :

The focal autopkgtest regressions were all resolved by retries.

Revision history for this message
Mustafa Kemal Gilor (mustafakemalgilor) wrote :
Download full text (3.1 KiB)

Verification done for focal:

- Environment -

ubuntu@crustle:~$ uname -a
Linux crustle 5.4.0-125-generic #141-Ubuntu SMP Wed Aug 10 13:42:03 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

ubuntu@crustle:~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

ubuntu@crustle:~$ systemd --version
systemd 245 (245.4-4ubuntu3.18)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid

root@crustle:/home/ubuntu# cat /sys/module/pstore/parameters/backend
efi

-----------------

- Test steps -

See [Test Plan]

-----------------

- Result -

Verification : OK

root@crustle:/home/ubuntu# echo 1 > /proc/sys/kernel/sysrq
root@crustle:/home/ubuntu# echo 1 > /proc/sys/kernel/panic
root@crustle:/home/ubuntu# echo "c" > /proc/sysrq-trigger

* system reboots *

root@crustle:/home/ubuntu# ls /sys/fs/pstore
root@crustle:/home/ubuntu# ls /var/lib/systemd/pstore
166271364
root@crustle:/home/ubuntu# systemctl status systemd-pstore
● systemd-pstore.service - Platform Persistent Storage Archival
     Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled)
     Active: active (exited) since Fri 2022-09-09 08:57:29 UTC; 3min 10s ago
       Docs: man:systemd-pstore(8)
    Process: 639 ExecStart=/lib/systemd/systemd-pstore (code=exited, status=0/SUCCESS)
   Main PID: 639 (code=exited, status=0/SUCCESS)

Sep 09 08:57:29 crustle systemd-pstore[639]: PStore dmesg-efi-166271364709001 moved to /var/lib/systemd/pstore/166271364/dmesg-efi-166271364709001
Sep 09 08:57:29 crustle systemd-pstore[639]: PStore dmesg-efi-166271364708001 moved to /var/lib/systemd/pstore/166271364/dmesg-efi-166271364708001
Sep 09 08:57:29 crustle systemd-pstore[639]: PStore dmesg-efi-166271364607001 moved to /var/lib/systemd/pstore/166271364/dmesg-efi-166271364607001
Sep 09 08:57:29 crustle systemd-pstore[639]: PStore dmesg-efi-166271364606001 moved to /var/lib/systemd/pstore/166271364/dmesg-efi-166271364606001
Sep 09 08:57:29 crustle systemd-pstore[639]: PStore dmesg-efi-166271364605001 moved to /var/lib/systemd/pstore/166271364/dmesg-efi-166271364605001
Sep 09 08:57:29 crustle systemd-pstore[639]: PStore dmesg-efi-166271364604001 moved to /var/lib/systemd/pstore/166271364/dmesg-efi-166271364604001
Sep 09 08:57:29 crustle systemd-pstore[639]: PStore dmesg-efi-166271364603001 moved to /var/lib/systemd/pstore/166271364/dmesg-efi-166271364603001
Sep 09 08:57:29 crustle systemd-pstore[639]: PStore dmesg-efi-166271364602001 moved to /var/lib/systemd/pstore/166271364/dmesg-efi-166271364602001
Sep 09 08:57:29 crustle systemd-pstore[639]: PStore dmesg-efi-166271364601001 moved to /var/lib/systemd/pstore/166271364/dmesg-efi-166271364601001
Sep 09 08:57:29 crustle systemd[1]: Finished Platform Persistent Storage...

Read more...

tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 245.4-4ubuntu3.18

---------------
systemd (245.4-4ubuntu3.18) focal; urgency=medium

  [ Nick Rosbrook ]
  * core: make sure we don't get confused when setting TERM for a tty fd
    (LP: #1959475)
    File: debian/patches/lp1959475-core-make-sure-we-don-t-get-confused-when-setting-TERM-fo.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=b10c6853050dde26665caf3b15444d768d2bc498
  * shared/calendarspec: when mktime() moves us backwards, jump forward
    (LP: #1966800)
    File: debian/patches/lp1966800-shared-calendarspec-when-mktime-moves-us-backwards-jump-f.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=1f063541e44f6ff1a6904676d4264a2e49a09594
  * network: do not remove localhost address (LP: #1979951)
    File: debian/patches/lp1979951-network-do-not-remove-localhost-address.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=87f872b8c5451f353601fb606e7fd7a479217cef
  * units: remove the restart limit on the modprobe@.service (LP: #1982462)
    File: debian/patches/lp1982462-units-remove-the-restart-limit-on-the-modprobe-.service.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=68353ffaf3539e6a58ef62a8b50850f56eae29ea

  [ Mustafa Kemal Gilor ]
  * d/p/lp1978079-efi-pstore-not-cleared-on-boot.patch: pstore: Run after
    modules are loaded. Thanks to Alexander Graf <email address hidden>.
    (LP: #1978079)
    Author: Mustafa Kemal Gilor
    File: debian/patches/lp1978079-efi-pstore-not-cleared-on-boot.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=6e60756f2079d6408abdb967127a1d9b9a0eba8c

 -- Nick Rosbrook <email address hidden> Wed, 31 Aug 2022 11:27:33 -0400

Changed in systemd (Ubuntu Focal):
status: Fix Committed → Fix Released
William Tu (wtu)
no longer affects: linux-bluefield (Ubuntu Impish)
no longer affects: linux-bluefield (Ubuntu Kinetic)
Changed in linux-bluefield (Ubuntu):
assignee: nobody → William Tu (wtu)
Changed in linux-bluefield (Ubuntu Focal):
assignee: nobody → William Tu (wtu)
Changed in linux-bluefield (Ubuntu Jammy):
assignee: nobody → William Tu (wtu)
Changed in linux-bluefield (Ubuntu Jammy):
status: New → Fix Committed
Revision history for this message
Bartlomiej Zolnierkiewicz (bzolnier) wrote :

This bug is awaiting verification that the linux-bluefield/5.15.0-1022.24 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-jammy
removed: verification-done-jammy
Revision history for this message
Feysel Mohammed (feyselm) wrote (last edit ):

root@bu-lab24v-oob:~# cat /etc/mlnx-release
DOCA_v2.2.0_BlueField_OS_4.2.1_Ubuntu_22.04-1.sru-5.15.0-1022
root@bu-lab24v-oob:~# uname -a
Linux bu-lab24v-oob 5.15.0-1022-bluefield #24-Ubuntu SMP Mon Aug 21 09:02:52 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
root@bu-lab24v-oob:~# cat /sys/module/pstore/parameters/backend
efi

root@bu-lab24v-oob:~# echo 1 > /proc/sys/kernel/sysrq
root@bu-lab24v-oob:~# echo 1 > /proc/sys/kernel/panic
root@bu-lab24v-oob:~# echo "c" > /proc/sysrq-trigger

*system rebooted*

root@bu-lab24v-oob:~# ls /sys/fs/pstore
root@bu-lab24v-oob:~# ls /var/lib/systemd/pstore
169289837
root@bu-lab24v-oob:~# systemctl status systemd-pstore.service
● systemd-pstore.service - Platform Persistent Storage Archival
     Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled)
     Active: active (exited) since Thu 2023-08-24 17:34:02 UTC; 6min ago
       Docs: man:systemd-pstore(8)
    Process: 560 ExecStart=/lib/systemd/systemd-pstore (code=exited, status=0/SUCCESS)
   Main PID: 560 (code=exited, status=0/SUCCESS)
        CPU: 21ms

Aug 24 17:34:01 localhost systemd-pstore[560]: PStore dmesg-efi-169289837309001 moved to /var/lib/systemd/pstore/169289837/dmesg-efi-1692898>
Aug 24 17:34:01 localhost systemd-pstore[560]: PStore dmesg-efi-169289837308001 moved to /var/lib/systemd/pstore/169289837/dmesg-efi-1692898>
Aug 24 17:34:01 localhost systemd-pstore[560]: PStore dmesg-efi-169289837307001 moved to /var/lib/systemd/pstore/169289837/dmesg-efi-1692898>
Aug 24 17:34:02 localhost systemd-pstore[560]: PStore dmesg-efi-169289837306001 moved to /var/lib/systemd/pstore/169289837/dmesg-efi-1692898>
Aug 24 17:34:02 localhost systemd-pstore[560]: PStore dmesg-efi-169289837305001 moved to /var/lib/systemd/pstore/169289837/dmesg-efi-1692898>
Aug 24 17:34:02 localhost systemd-pstore[560]: PStore dmesg-efi-169289837304001 moved to /var/lib/systemd/pstore/169289837/dmesg-efi-1692898>
Aug 24 17:34:02 localhost systemd-pstore[560]: PStore dmesg-efi-169289837303001 moved to /var/lib/systemd/pstore/169289837/dmesg-efi-1692898>
Aug 24 17:34:02 localhost systemd-pstore[560]: PStore dmesg-efi-169289837302001 moved to /var/lib/systemd/pstore/169289837/dmesg-efi-1692898>
Aug 24 17:34:02 localhost systemd-pstore[560]: PStore dmesg-efi-169289837301001 moved to /var/lib/systemd/pstore/169289837/dmesg-efi-1692898>
Aug 24 17:34:02 localhost systemd[1]: Finished Platform Persistent Storage Archival.

tags: added: verification-done-jammy
removed: verification-needed-jammy
Agathe Porte (gagath)
Changed in linux-bluefield (Ubuntu Focal):
status: New → Fix Committed
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-bluefield/5.4.0-1070.76 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal-linux-bluefield' to 'verification-done-focal-linux-bluefield'. If the problem still exists, change the tag 'verification-needed-focal-linux-bluefield' to 'verification-failed-focal-linux-bluefield'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-focal-linux-bluefield-v2 verification-needed-focal-linux-bluefield
Revision history for this message
Feysel Mohammed (feyselm) wrote :

root@bu-lab26v-oob:~# cat /etc/mlnx-release
DOCA_2.2.0_BSP_4.2.1_Ubuntu_20.04-2.sru.5.4.0-1070
root@bu-lab26v-oob:~# uname -a
Linux bu-lab26v-oob 5.4.0-1070-bluefield #76-Ubuntu SMP PREEMPT Wed Aug 30 16:56:35 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
root@bu-lab26v-oob:~# cat /sys/module/pstore/parameters/backend
efi
root@bu-lab26v-oob:~# systemctl status systemd-pstore.service
● systemd-pstore.service - Platform Persistent Storage Archival
     Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
  Condition: start condition failed at Tue 2023-09-19 14:16:32 UTC; 1 day 1h ago
             └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met
       Docs: man:systemd-pstore(8)

Sep 19 14:16:28 localhost systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped.
Sep 19 14:16:32 localhost systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped.
root@bu-lab26v-oob:~# echo 1 > /proc/sys/kernel/sysrq
root@bu-lab26v-oob:~# echo 1 > /proc/sys/kernel/panic
root@bu-lab26v-oob:~# echo "c" > /proc/sysrq-trigger

*system rebooted*

root@bu-lab26v-oob:~# ls /sys/fs/pstore
root@bu-lab26v-oob:~# ls /var/lib/systemd/pstore
169522441 169522442
root@bu-lab26v-oob:~# systemctl status systemd-pstore.service
● systemd-pstore.service - Platform Persistent Storage Archival
     Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled)
     Active: active (exited) since Wed 2023-09-20 15:41:29 UTC; 56s ago
       Docs: man:systemd-pstore(8)
    Process: 485 ExecStart=/lib/systemd/systemd-pstore (code=exited, status=0/SUCCESS)
   Main PID: 485 (code=exited, status=0/SUCCESS)

Sep 20 15:41:29 localhost systemd-pstore[485]: PStore dmesg-efi-169522441409001 moved to /var/lib/systemd/pstore/169522441/dmesg-efi-169522441409001
Sep 20 15:41:29 localhost systemd-pstore[485]: PStore dmesg-efi-169522441408001 moved to /var/lib/systemd/pstore/169522441/dmesg-efi-169522441408001
Sep 20 15:41:29 localhost systemd-pstore[485]: PStore dmesg-efi-169522441407001 moved to /var/lib/systemd/pstore/169522441/dmesg-efi-169522441407001
Sep 20 15:41:29 localhost systemd-pstore[485]: PStore dmesg-efi-169522441406001 moved to /var/lib/systemd/pstore/169522441/dmesg-efi-169522441406001
Sep 20 15:41:29 localhost systemd-pstore[485]: PStore dmesg-efi-169522441405001 moved to /var/lib/systemd/pstore/169522441/dmesg-efi-169522441405001
Sep 20 15:41:29 localhost systemd-pstore[485]: PStore dmesg-efi-169522441304001 moved to /var/lib/systemd/pstore/169522441/dmesg-efi-169522441304001
Sep 20 15:41:29 localhost systemd-pstore[485]: PStore dmesg-efi-169522441303001 moved to /var/lib/systemd/pstore/169522441/dmesg-efi-169522441303001
Sep 20 15:41:29 localhost systemd-pstore[485]: PStore dmesg-efi-169522441302001 moved to /var/lib/systemd/pstore/169522441/dmesg-efi-169522441302001
Sep 20 15:41:29 localhost systemd-pstore[485]: PStore dmesg-efi-169522441301001 moved to /var/lib/systemd/pstore/169522441/dmesg-efi-169522441301001
Sep 20 15:41:29 localhost systemd[1]: Finished Platform Persistent Storage Archival.

tags: added: verification-done-focal-linux-bluefield
removed: verification-needed-focal-linux-bluefield
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (40.3 KiB)

This bug was fixed in the package linux-bluefield - 5.4.0-1070.76

---------------
linux-bluefield (5.4.0-1070.76) focal; urgency=medium

  * focal/linux-bluefield: 5.4.0-1070.76 -proposed tracker (LP: #2030986)

  * EFI pstore not cleared on boot (LP: #1978079)
    - [Config] bluefield: Enable CONFIG_EFI_VARS_PSTORE as build-in
    - [Config] bluefield: Remove efi-pstore from modules list

  * Focal update: v5.4.246 upstream stable release (LP: #2028981)
    - [Config] bluefield: updateconfigs for SCSI_DPT_I2O
    - SAUCE: xfrm: Check if_id in xfrm full offload

  * rshim console truncates dmesg output due to tmfifo issue (LP: #2028197)
    - SAUCE: mlxbf-tmfifo.c: Fix rhsim console w/ truncated dmesg output

  * mlxbf-gige: Fix kernel panic after reboot (LP: #2030765)
    - SAUCE: mlxbf-gige: Fix kernel panic after reboot (part 1/2)
    - SAUCE: mlxbf-gige: Fix kernel panic after reboot (part 2/2)

  [ Ubuntu: 5.4.0-162.179 ]

  * focal/linux: 5.4.0-162.179 -proposed tracker (LP: #2031128)
  * libgnutls report "trap invalid opcode" when trying to install packages over
    https (LP: #2031093)
    - [Config]: disable CONFIG_GDS_FORCE_MITIGATION

  [ Ubuntu: 5.4.0-160.177 ]

  * focal/linux: 5.4.0-160.177 -proposed tracker (LP: #2031017)
  * Fix boot test warning for log_check "CPU: 0 PID: 0 at
    arch/x86/kernel/fpu/xstate.c:878 get_xsave_addr+0x98/0xb0" (LP: #2031022)
    - x86/pkeys: Revert a5eff7259790 ("x86/pkeys: Add PKRU value to init_fpstate")

  [ Ubuntu: 5.4.0-158.175 ]

  * focal/linux: 5.4.0-158.175 -proposed tracker (LP: #2030466)
  * CVE-2022-40982
    - x86/mm: Initialize text poking earlier
    - x86/mm: fix poking_init() for Xen PV guests
    - x86/mm: Use mm_alloc() in poking_init()
    - mm: Move mm_cachep initialization to mm_init()
    - init: Provide arch_cpu_finalize_init()
    - x86/cpu: Switch to arch_cpu_finalize_init()
    - ARM: cpu: Switch to arch_cpu_finalize_init()
    - sparc/cpu: Switch to arch_cpu_finalize_init()
    - um/cpu: Switch to arch_cpu_finalize_init()
    - init: Remove check_bugs() leftovers
    - init: Invoke arch_cpu_finalize_init() earlier
    - init, x86: Move mem_encrypt_init() into arch_cpu_finalize_init()
    - x86/fpu: Remove cpuinfo argument from init functions
    - x86/fpu: Mark init functions __init
    - x86/fpu: Move FPU initialization into arch_cpu_finalize_init()
    - x86/xen: Fix secondary processors' FPU initialization
    - x86/speculation: Add Gather Data Sampling mitigation
    - x86/speculation: Add force option to GDS mitigation
    - x86/speculation: Add Kconfig option for GDS
    - KVM: Add GDS_NO support to KVM
    - Documentation/x86: Fix backwards on/off logic about YMM support
    - [Config]: Enable CONFIG_ARCH_HAS_CPU_FINALIZE_INIT and
      CONFIG_GDS_FORCE_MITIGATION
  * CVE-2023-3609
    - net/sched: cls_u32: Fix reference counter leak leading to overflow
  * CVE-2023-20593
    - x86/cpu/amd: Move the errata checking functionality up
    - x86/cpu/amd: Add a Zenbleed fix
  * CVE-2023-3611
    - net/sched: sch_qfq: account for stab overhead in qfq_enqueue
  * stacked overlay file system mounts that have chroot() called against them
    appear to be getting locked (...

Changed in linux-bluefield (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.