Activity log for bug #1978079

Date Who What changed Old value New value Message
2022-06-09 09:44:12 Alexander Graf bug added bug
2022-06-09 13:57:23 Dan Streetman bug added subscriber Dan Streetman
2022-06-09 14:07:26 Alexander Graf bug watch added https://github.com/systemd/systemd/issues/18540
2022-06-14 13:17:55 Nick Rosbrook tags efi pstore uefi efi pstore rls-ff-incoming rls-jj-incoming uefi
2022-06-16 06:51:34 Mustafa Kemal Gilor systemd (Ubuntu): assignee Mustafa Kemal Gilor (mustafakemalgilor)
2022-06-16 06:51:43 Mustafa Kemal Gilor systemd (Ubuntu): status New In Progress
2022-06-16 15:28:25 Matthieu Clemenceau tags efi pstore rls-ff-incoming rls-jj-incoming uefi efi fr-2484 pstore rls-ff-incoming rls-jj-incoming uefi
2022-06-16 15:37:34 Brian Murray tags efi fr-2484 pstore rls-ff-incoming rls-jj-incoming uefi efi fr-2484 pstore uefi
2022-06-17 09:33:50 Launchpad Janitor merge proposal linked https://code.launchpad.net/~mustafakemalgilor/ubuntu/+source/systemd/+git/systemd/+merge/424938
2022-06-17 12:55:15 Launchpad Janitor merge proposal linked https://code.launchpad.net/~mustafakemalgilor/ubuntu/+source/systemd/+git/systemd/+merge/424958
2022-06-20 07:36:15 Mustafa Kemal Gilor tags efi fr-2484 pstore uefi efi fr-2484 pstore seg sts sts-sponsor uefi
2022-06-20 09:48:12 Mustafa Kemal Gilor tags efi fr-2484 pstore seg sts sts-sponsor uefi efi fr-2484 pstore seg sts sts-sponsor ubuntu-sponsors uefi
2022-06-20 10:11:54 Mustafa Kemal Gilor description Systemd has a systemd-pstore component that scans the pstore on boot and if non-empty, takes all previously created dumps, transfers them into its journal and removes the pstore elements. This is very important on UEFI systems, which only have a limited amount of space for variables. In Ubuntu, the kernel is configured with CONFIG_EFI_VARS_PSTORE=m which means the EFI pstore support gets loaded dynamically. In all of my boots, this dynamic module loading happened *after* systemd tried to check for pstore variables. So systemd-pstore never starts and never clears the UEFI variable store. I see this happening in AWS on Graviton instances, which eventually run out of space to store the dumps. On real hardware, this behavior may lead to unbootable systems. ``` $ systemctl status systemd-pstore ○ systemd-pstore.service - Platform Persistent Storage Archival Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met Docs: man:systemd-pstore(8) Jun 09 09:11:41 ip-172-31-0-61 systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped. $ ls -la /sys/fs/pstore total 0 drwxr-x--- 2 root root 0 Jun 9 09:11 . drwxr-xr-x 8 root root 0 Jun 9 09:11 .. -r--r--r-- 1 root root 1803 Jun 9 09:07 dmesg-efi-165476562001001 -r--r--r-- 1 root root 1777 Jun 9 09:07 dmesg-efi-165476562002001 -r--r--r-- 1 root root 1773 Jun 9 09:07 dmesg-efi-165476562003001 -r--r--r-- 1 root root 1815 Jun 9 09:07 dmesg-efi-165476562004001 -r--r--r-- 1 root root 1826 Jun 9 09:07 dmesg-efi-165476562005001 -r--r--r-- 1 root root 1754 Jun 9 09:07 dmesg-efi-165476562006001 -r--r--r-- 1 root root 1821 Jun 9 09:07 dmesg-efi-165476562007001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562008001 -r--r--r-- 1 root root 1729 Jun 9 09:07 dmesg-efi-165476562009001 -r--r--r-- 1 root root 1819 Jun 9 09:07 dmesg-efi-165476562010001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562011001 -r--r--r-- 1 root root 1775 Jun 9 09:07 dmesg-efi-165476562012001 -r--r--r-- 1 root root 1802 Jun 9 09:07 dmesg-efi-165476562013001 -r--r--r-- 1 root root 1812 Jun 9 09:07 dmesg-efi-165476562014001 -r--r--r-- 1 root root 1764 Jun 9 09:07 dmesg-efi-165476562015001 -r--r--r-- 1 root root 1795 Jun 9 09:11 dmesg-efi-165476589801001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589802001 -r--r--r-- 1 root root 1683 Jun 9 09:11 dmesg-efi-165476589803001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589804001 -r--r--r-- 1 root root 1771 Jun 9 09:11 dmesg-efi-165476589805001 -r--r--r-- 1 root root 1797 Jun 9 09:11 dmesg-efi-165476589806001 -r--r--r-- 1 root root 1805 Jun 9 09:11 dmesg-efi-165476589807001 -r--r--r-- 1 root root 1781 Jun 9 09:11 dmesg-efi-165476589808001 -r--r--r-- 1 root root 1806 Jun 9 09:11 dmesg-efi-165476589809001 -r--r--r-- 1 root root 1821 Jun 9 09:11 dmesg-efi-165476589810001 -r--r--r-- 1 root root 1763 Jun 9 09:11 dmesg-efi-165476589811001 -r--r--r-- 1 root root 1783 Jun 9 09:11 dmesg-efi-165476589812001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589813001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589814001 -r--r--r-- 1 root root 1786 Jun 9 09:11 dmesg-efi-165476589815001 ``` This problem affects (at least) Ubuntu 20.04 and 22.04. A quick fix would be to configure CONFIG_EFI_VARS_PSTORE=y so that it's always available. A long term fix would make systemd rescan the directory after all module probing settled. [Impact] Systemd has a systemd-pstore component that scans the pstore on boot and if non-empty, takes all previously created dumps, transfers them into its journal and removes the pstore elements. This is very important on UEFI systems, which only have a limited amount of space for variables. In Ubuntu, the kernel is configured with CONFIG_EFI_VARS_PSTORE=m which means the EFI pstore support gets loaded dynamically. In all of my boots, this dynamic module loading happened *after* systemd tried to check for pstore variables. So systemd-pstore never starts and never clears the UEFI variable store. I see this happening in AWS on Graviton instances, which eventually run out of space to store the dumps. On real hardware, this behavior may lead to unbootable systems. ``` $ systemctl status systemd-pstore ○ systemd-pstore.service - Platform Persistent Storage Archival      Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled)      Active: inactive (dead)   Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago              └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met        Docs: man:systemd-pstore(8) Jun 09 09:11:41 ip-172-31-0-61 systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped. $ ls -la /sys/fs/pstore total 0 drwxr-x--- 2 root root 0 Jun 9 09:11 . drwxr-xr-x 8 root root 0 Jun 9 09:11 .. -r--r--r-- 1 root root 1803 Jun 9 09:07 dmesg-efi-165476562001001 -r--r--r-- 1 root root 1777 Jun 9 09:07 dmesg-efi-165476562002001 -r--r--r-- 1 root root 1773 Jun 9 09:07 dmesg-efi-165476562003001 -r--r--r-- 1 root root 1815 Jun 9 09:07 dmesg-efi-165476562004001 -r--r--r-- 1 root root 1826 Jun 9 09:07 dmesg-efi-165476562005001 -r--r--r-- 1 root root 1754 Jun 9 09:07 dmesg-efi-165476562006001 -r--r--r-- 1 root root 1821 Jun 9 09:07 dmesg-efi-165476562007001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562008001 -r--r--r-- 1 root root 1729 Jun 9 09:07 dmesg-efi-165476562009001 -r--r--r-- 1 root root 1819 Jun 9 09:07 dmesg-efi-165476562010001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562011001 -r--r--r-- 1 root root 1775 Jun 9 09:07 dmesg-efi-165476562012001 -r--r--r-- 1 root root 1802 Jun 9 09:07 dmesg-efi-165476562013001 -r--r--r-- 1 root root 1812 Jun 9 09:07 dmesg-efi-165476562014001 -r--r--r-- 1 root root 1764 Jun 9 09:07 dmesg-efi-165476562015001 -r--r--r-- 1 root root 1795 Jun 9 09:11 dmesg-efi-165476589801001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589802001 -r--r--r-- 1 root root 1683 Jun 9 09:11 dmesg-efi-165476589803001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589804001 -r--r--r-- 1 root root 1771 Jun 9 09:11 dmesg-efi-165476589805001 -r--r--r-- 1 root root 1797 Jun 9 09:11 dmesg-efi-165476589806001 -r--r--r-- 1 root root 1805 Jun 9 09:11 dmesg-efi-165476589807001 -r--r--r-- 1 root root 1781 Jun 9 09:11 dmesg-efi-165476589808001 -r--r--r-- 1 root root 1806 Jun 9 09:11 dmesg-efi-165476589809001 -r--r--r-- 1 root root 1821 Jun 9 09:11 dmesg-efi-165476589810001 -r--r--r-- 1 root root 1763 Jun 9 09:11 dmesg-efi-165476589811001 -r--r--r-- 1 root root 1783 Jun 9 09:11 dmesg-efi-165476589812001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589813001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589814001 -r--r--r-- 1 root root 1786 Jun 9 09:11 dmesg-efi-165476589815001 ``` This problem affects (at least) Ubuntu 20.04 and 22.04. A quick fix would be to configure CONFIG_EFI_VARS_PSTORE=y so that it's always available. A long term fix would make systemd rescan the directory after all module probing settled. [Test Plan] In order to be able to reproduce this issue, the system must have EFI-backed pstore. To check which kind of backend that pstore, use `cat /sys/module/pstore/parameters/backend` If it says `efi`, the steps below are applicable. Otherwise, find an environment that has EFI backed pstore. # Enable the pstore service. This service is supposed to move the data in /sys/fs/pstore # to the `/var/lib/systemd/pstore` path on boot. systemctl enable systemd-pstore.service # (or can be vendor enabled) # Crash the kernel echo 1 > /proc/sys/kernel/sysrq echo 1 > /proc/sys/kernel/panic # this is usually set to zero, causing kernel to loop over the panic and freeze echo "c" > /proc/sysrq-trigger # The system will reboot itself. Check `/sys/fs/pstore` path first: ls /sys/fs/pstore # The path should not be empty, which means the systemd-pstore has failed to do its' job ls /var/lib/systemd/pstore # The path should be empty. # Apply the fix sudo add-apt-repository ppa:mustafakemalgilor/lp-1978079-1 sudo apt upgrade # Crash the kernel echo 1 > /proc/sys/kernel/sysrq echo 1 > /proc/sys/kernel/panic # this is usually set to zero, causing kernel to loop over the panic and freeze echo "c" > /proc/sysrq-trigger # The system will reboot itself. After reboot, the contents of the `/sys/fs/pstore` must have been moved to the `/var/lib/systemd/pstore` path. ls /sys/fs/pstore # The path should be empty ls /var/lib/systemd/pstore # The path should not be empty [Where problems could occur] On some systems, even though the described bug is present, the effect of the bug could not be observed. The nature of the issue suggests that this is a due to a timing issue.
2022-06-20 11:54:39 Launchpad Janitor merge proposal linked https://code.launchpad.net/~mustafakemalgilor/ubuntu/+source/systemd/+git/systemd/+merge/425077
2022-06-20 12:56:23 Launchpad Janitor merge proposal linked https://code.launchpad.net/~mustafakemalgilor/ubuntu/+source/systemd/+git/systemd/+merge/425081
2022-06-20 12:59:39 Mustafa Kemal Gilor merge proposal unlinked https://code.launchpad.net/~mustafakemalgilor/ubuntu/+source/systemd/+git/systemd/+merge/425077
2022-06-20 13:01:29 Launchpad Janitor merge proposal linked https://code.launchpad.net/~mustafakemalgilor/ubuntu/+source/systemd/+git/systemd/+merge/425082
2022-06-20 13:01:32 Launchpad Janitor merge proposal linked https://code.launchpad.net/~mustafakemalgilor/ubuntu/+source/systemd/+git/systemd/+merge/425083
2022-06-20 13:01:53 Mustafa Kemal Gilor merge proposal unlinked https://code.launchpad.net/~mustafakemalgilor/ubuntu/+source/systemd/+git/systemd/+merge/424958
2022-06-20 13:02:10 Mustafa Kemal Gilor merge proposal unlinked https://code.launchpad.net/~mustafakemalgilor/ubuntu/+source/systemd/+git/systemd/+merge/424938
2022-06-21 14:08:42 Dariusz Gadomski tags efi fr-2484 pstore seg sts sts-sponsor ubuntu-sponsors uefi efi fr-2484 pstore seg sts sts-sponsor sts-sponsor-dgadomski ubuntu-sponsors uefi
2022-06-21 14:39:18 Launchpad Janitor merge proposal linked https://code.launchpad.net/~enr0n/ubuntu/+source/systemd/+git/systemd/+merge/425169
2022-06-21 14:40:14 Launchpad Janitor merge proposal linked https://code.launchpad.net/~enr0n/ubuntu/+source/systemd/+git/systemd/+merge/425171
2022-06-23 14:42:31 Lukas Märdian nominated for series Ubuntu Impish
2022-06-23 14:42:31 Lukas Märdian bug task added systemd (Ubuntu Impish)
2022-06-23 14:42:31 Lukas Märdian nominated for series Ubuntu Kinetic
2022-06-23 14:42:31 Lukas Märdian bug task added systemd (Ubuntu Kinetic)
2022-06-23 14:42:31 Lukas Märdian nominated for series Ubuntu Focal
2022-06-23 14:42:31 Lukas Märdian bug task added systemd (Ubuntu Focal)
2022-06-23 14:42:31 Lukas Märdian nominated for series Ubuntu Jammy
2022-06-23 14:42:31 Lukas Märdian bug task added systemd (Ubuntu Jammy)
2022-06-23 14:42:50 Lukas Märdian systemd (Ubuntu Jammy): status New In Progress
2022-06-23 14:42:52 Lukas Märdian systemd (Ubuntu Focal): status New In Progress
2022-06-23 14:43:27 Lukas Märdian systemd (Ubuntu Focal): assignee Mustafa Kemal Gilor (mustafakemalgilor)
2022-06-23 14:43:32 Lukas Märdian systemd (Ubuntu Jammy): assignee Mustafa Kemal Gilor (mustafakemalgilor)
2022-06-23 14:43:37 Lukas Märdian systemd (Ubuntu Kinetic): assignee Mustafa Kemal Gilor (mustafakemalgilor) Nick Rosbrook (enr0n)
2022-06-23 16:15:57 Lukas Märdian systemd (Ubuntu Impish): status New Won't Fix
2022-06-28 09:15:45 Łukasz Zemczak systemd (Ubuntu Jammy): status In Progress Fix Committed
2022-06-28 09:15:47 Łukasz Zemczak bug added subscriber Ubuntu Stable Release Updates Team
2022-06-28 09:15:49 Łukasz Zemczak bug added subscriber SRU Verification
2022-06-28 09:15:52 Łukasz Zemczak tags efi fr-2484 pstore seg sts sts-sponsor sts-sponsor-dgadomski ubuntu-sponsors uefi efi fr-2484 pstore seg sts sts-sponsor sts-sponsor-dgadomski ubuntu-sponsors uefi verification-needed verification-needed-jammy
2022-06-30 08:26:44 Lukas Märdian systemd (Ubuntu Kinetic): status In Progress Fix Committed
2022-07-01 13:57:48 Dariusz Gadomski bug added subscriber Dariusz Gadomski
2022-07-02 16:58:47 Launchpad Janitor systemd (Ubuntu Kinetic): status Fix Committed Fix Released
2022-07-02 16:58:47 Launchpad Janitor bug watch added https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=746463
2022-07-02 16:58:47 Launchpad Janitor bug watch added https://github.com/systemd/systemd/issues/22635
2022-07-02 16:58:47 Launchpad Janitor bug watch added https://github.com/systemd/systemd/issues/22991
2022-07-02 16:58:47 Launchpad Janitor bug watch added https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=969568
2022-07-02 16:58:47 Launchpad Janitor bug watch added https://github.com/systemd/systemd/issues/21975
2022-07-02 16:58:47 Launchpad Janitor cve linked 2021-3997
2022-07-05 07:10:16 Mustafa Kemal Gilor tags efi fr-2484 pstore seg sts sts-sponsor sts-sponsor-dgadomski ubuntu-sponsors uefi verification-needed verification-needed-jammy efi fr-2484 pstore seg sts sts-sponsor sts-sponsor-dgadomski ubuntu-sponsors uefi verification-done-jammy verification-needed
2022-07-05 20:42:18 Launchpad Janitor systemd (Ubuntu Jammy): status Fix Committed Fix Released
2022-07-05 20:42:28 Brian Murray removed subscriber Ubuntu Stable Release Updates Team
2022-09-05 13:10:51 Łukasz Zemczak systemd (Ubuntu Focal): status In Progress Fix Committed
2022-09-05 13:10:54 Łukasz Zemczak bug added subscriber Ubuntu Stable Release Updates Team
2022-09-05 13:10:58 Łukasz Zemczak tags efi fr-2484 pstore seg sts sts-sponsor sts-sponsor-dgadomski ubuntu-sponsors uefi verification-done-jammy verification-needed efi fr-2484 pstore seg sts sts-sponsor sts-sponsor-dgadomski ubuntu-sponsors uefi verification-done-jammy verification-needed verification-needed-focal
2022-09-06 10:20:53 Launchpad Janitor merge proposal linked https://code.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/+git/systemd/+merge/429490
2022-09-09 09:04:31 Mustafa Kemal Gilor tags efi fr-2484 pstore seg sts sts-sponsor sts-sponsor-dgadomski ubuntu-sponsors uefi verification-done-jammy verification-needed verification-needed-focal efi fr-2484 pstore seg sts sts-sponsor sts-sponsor-dgadomski ubuntu-sponsors uefi verification-done-focal verification-done-jammy verification-needed
2022-09-22 09:15:24 Launchpad Janitor systemd (Ubuntu Focal): status Fix Committed Fix Released
2023-07-26 15:54:06 William Tu bug task added linux-bluefield (Ubuntu)
2023-07-26 15:54:33 William Tu bug task deleted linux-bluefield (Ubuntu Impish)
2023-07-26 15:54:41 William Tu bug task deleted linux-bluefield (Ubuntu Kinetic)
2023-07-26 15:55:46 William Tu linux-bluefield (Ubuntu): assignee William Tu (wtu)
2023-07-26 15:55:48 William Tu linux-bluefield (Ubuntu Focal): assignee William Tu (wtu)
2023-07-26 15:55:50 William Tu linux-bluefield (Ubuntu Jammy): assignee William Tu (wtu)
2023-07-26 16:25:27 Dan Streetman removed subscriber Dan Streetman
2023-08-17 17:00:23 Bartlomiej Zolnierkiewicz linux-bluefield (Ubuntu Jammy): status New Fix Committed
2023-08-24 09:11:28 Bartlomiej Zolnierkiewicz tags efi fr-2484 pstore seg sts sts-sponsor sts-sponsor-dgadomski ubuntu-sponsors uefi verification-done-focal verification-done-jammy verification-needed efi fr-2484 pstore seg sts sts-sponsor sts-sponsor-dgadomski ubuntu-sponsors uefi verification-done-focal verification-needed verification-needed-jammy
2023-08-24 17:50:15 Feysel Mohammed tags efi fr-2484 pstore seg sts sts-sponsor sts-sponsor-dgadomski ubuntu-sponsors uefi verification-done-focal verification-needed verification-needed-jammy efi fr-2484 pstore seg sts sts-sponsor sts-sponsor-dgadomski ubuntu-sponsors uefi verification-done-focal verification-done-jammy verification-needed
2023-08-29 09:39:27 Agathe Porte linux-bluefield (Ubuntu Focal): status New Fix Committed
2023-08-30 23:46:17 Ubuntu Kernel Bot tags efi fr-2484 pstore seg sts sts-sponsor sts-sponsor-dgadomski ubuntu-sponsors uefi verification-done-focal verification-done-jammy verification-needed efi fr-2484 kernel-spammed-focal-linux-bluefield-v2 pstore seg sts sts-sponsor sts-sponsor-dgadomski ubuntu-sponsors uefi verification-done-focal verification-done-jammy verification-needed verification-needed-focal-linux-bluefield
2023-09-20 18:44:45 Feysel Mohammed tags efi fr-2484 kernel-spammed-focal-linux-bluefield-v2 pstore seg sts sts-sponsor sts-sponsor-dgadomski ubuntu-sponsors uefi verification-done-focal verification-done-jammy verification-needed verification-needed-focal-linux-bluefield efi fr-2484 kernel-spammed-focal-linux-bluefield-v2 pstore seg sts sts-sponsor sts-sponsor-dgadomski ubuntu-sponsors uefi verification-done-focal verification-done-focal-linux-bluefield verification-done-jammy verification-needed
2023-09-21 19:29:43 Launchpad Janitor linux-bluefield (Ubuntu Focal): status Fix Committed Fix Released
2023-09-21 19:29:43 Launchpad Janitor cve linked 2022-40982
2023-09-21 19:29:43 Launchpad Janitor cve linked 2023-20593
2023-09-21 19:29:43 Launchpad Janitor cve linked 2023-2269
2023-09-21 19:29:43 Launchpad Janitor cve linked 2023-31084
2023-09-21 19:29:43 Launchpad Janitor cve linked 2023-3268
2023-09-21 19:29:43 Launchpad Janitor cve linked 2023-3609
2023-09-21 19:29:43 Launchpad Janitor cve linked 2023-3611
2023-09-21 19:29:43 Launchpad Janitor cve linked 2023-3776