initrdfail can result in resuming with different initrd images and hanging resume

This bug was fixed in the package grub2 - 2.04-1ubuntu46

---------------
grub2 (2.04-1ubuntu46) impish; urgency=medium

  * debian/grub-common.service: change type to oneshot, add wantedby
    sleep.target, after sleep.target. The service will now start after
    resume from hybernation. LP: #1929860
  * grub-initrd-fallback.service: add wantedby sleep.target, after
    sleep.target. The service will now start after resume from
    hybernation. LP: #1929860
  * cherrypick upstream fix to make armhf efi boot work. LP: #1788940
  * debian/rules: disable LTO. LP: #1922005
  * grub-initrd-fallback.service, debian/grub-common.service: only start
    units when booted with grub. Use presence of /boot/grub/grub.cfg as
    proxy. LP: #1925507
  * tests: patch qemu command to use ide-hd instead of the removed
    ide-drive.

 -- Dimitri John Ledkov <email address hidden> Fri, 16 Jul 2021 14:01:31 +0100

Hello Francis, or anyone else affected,

Accepted grub2 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.04-1ubuntu26.13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

@juliank

This does not appear to be a complete fix for the issue seen on AWS t2.nano. I see failures in about 5 - 10% of cases. There are two different symptoms:

 * grub-initrd-fallback.service sometimes runs right before the system hibernates (it runs after the hibernation request was sent to the VM and before it and before it fully hibernates).
 * grub-initrd-fallback.service sometimes runs after resume, but checking the status of `initrdfail` a few minutes later indicates it is still set (not sure if it was ever cleared).

Both of these were determined by checking the last active timestamp reported by 'systemctl status grub-initrd-fallback.service' and comparing this with timestamps generated by the hibernation test. If either situation occurs, the next hibernation/resume will fail.

I'm trying to collect more information.

Thank you Francis for checking up on this! In this case we'll probably not block on this re: our usual 20.04.3. Would that be fine if this gets fixed after the regular flavors of 20.04.3 are released?

@sil2100

Following up on our chat conversation. I actually thing this grub2 package update could be released. It is greatly improving the situation and I think is doing the right thing. But I think there may be other external factors in play that may result in these corner cases.

I can give a verification done.

I tested the grub2 update on a virtual machine and was able to reboot and login without any issues.

bdmurray@clean-focal-amd64:~$ dpkg -l | grep ii.*grub
ii grub-common 2.04-1ubuntu26.13 amd64 GRand Unified Bootloader (common files)
ii grub-gfxpayload-lists 0.7 amd64 GRUB gfxpayload blacklist
ii grub-pc 2.04-1ubuntu26.13 amd64 GRand Unified Bootloader, version 2 (PC/BI
OS version)
ii grub-pc-bin 2.04-1ubuntu26.13 amd64 GRand Unified Bootloader, version 2 (PC/BI
OS modules)
ii grub2-common 2.04-1ubuntu26.13 amd64 GRand Unified Bootloader (common files for
 version 2)

I also tested this update on an Intel NUC and I was able to reboot w/o any issues.

bdmurray@atom:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.2 LTS"
bdmurray@atom:~$ uname -a
Linux atom 5.4.0-81-generic #91-Ubuntu SMP Thu Jul 15 19:09:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
bdmurray@atom:~$ dpkg -l | grep ii.*grub
ii grub-common 2.04-1ubuntu26.13 amd64 GRand Unified Bootloader (common files)
ii grub-efi-amd64 2.04-1ubuntu44.2 amd64 GRand Unified Bootloader, version 2 (EFI-AMD64 version)
ii grub-efi-amd64-bin 2.04-1ubuntu44.2 amd64 GRand Unified Bootloader, version 2 (EFI-AMD64 modules)
ii grub-efi-amd64-signed 1.167.2+2.04-1ubuntu44.2 amd64 GRand Unified Bootloader, version 2 (EFI-AMD64 version, signed)
ii grub2-common 2.04-1ubuntu26.13 amd64 GRand Unified Bootloader (common files for version 2)
ii ubuntu-recovery-grub-hotkey 1.1columbia1 all Ubuntu Recovery Grub Hotkey Configuration

Adding verification-done as this does fix the problem in nearly all testing. I think there is a corner case or something specific to the testing which is causing these outliers.

This bug was fixed in the package grub2 - 2.04-1ubuntu26.13

---------------
grub2 (2.04-1ubuntu26.13) focal; urgency=medium

  [ Julian Andres Klode ]
  * unapply all patches, use gbp pq instead of git-dpm

  [ Dimitri John Ledkov ]
  * 10_linux: emit messages when initrdless boot is configured, attempted and
    fails triggering fallback. LP: #1901553
  * grub-common.service: port init.d script to systemd unit. Add warning
    message, when initrdless boot fails triggering fallback. LP: #1901553
  * debian/grub-common.service: change type to oneshot, add wantedby
    sleep.target, after sleep.target. The service will now start after resume
    from hybernation. (LP: #1929860)
  * grub-initrd-fallback.service: add wantedby sleep.target, after
    sleep.target. The service will now start after resume from hybernation.
    LP: #1929860
  * grub-initrd-fallback.service, debian/grub-common.service: only start units
    when booted with grub. Use presence of /boot/grub/grub.cfg as proxy. LP:
    #1925507

 -- Julian Andres Klode <email address hidden> Thu, 12 Aug 2021 11:18:25 +0200

The verification of the Stable Release Update for grub2 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

An upload of grub2 to bionic-proposed has been rejected from the upload queue for the following reason: "The debdiff for this is a mess, there is delta to all the packages under debian/patches, impossible to review; please reupload without the extra delta".