Ubuntu removes CentOS UEFI boot entry on shutdown

I'm not sure if the entry is deleted, or indeed, fails to be added.

I'm deploying onto nvme drive and whilst:

chroot /mnt efibootmgr -v --create --write-signature --label centos --disk /dev/nvme0n1 --part 1 --loader /boot/efi/EFI/centos/shimx64.efiCould not prepare Boot variable: No such file or directory

fails, (aka using efibootmgr from the chrooted system) whilst booted into ubuntu's v5.4 kernel.

But running the same command from focal environment (i.e. without chroot). Does succeed fine:

efibootmgr -v --create --write-signature --label centos --disk /dev/nvme0n1 --part 1 --loader /boot/efi/EFI/centos/shimx64.efi

$ sudo efibootmgr -v --create --write-signature --label centos --disk /dev/nvme0n1 --part 1 --loader /boot/efi/EFI/centos/shimx64.efi
BootNext: 0002
BootCurrent: 0002
Timeout: 2 seconds
BootOrder: 0000,0002,0003,0004,0005,0006,0001
Boot0001* UEFI: Built-in EFI Shell VenMedia(5023b95c-db26-429b-a648-bd47664c8012)..BO
Boot0002* UEFI: PXE IP4 Intel(R) I350 Gigabit Network Connection PciRoot(0x3)/Pci(0x3,0x4)/Pci(0x0,0x0)/MAC(18c04d711026,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot0003* UEFI: PXE IP4 Intel(R) I350 Gigabit Network Connection PciRoot(0x3)/Pci(0x3,0x4)/Pci(0x0,0x1)/MAC(18c04d711027,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot0004* UEFI: PXE IP6 Intel(R) I350 Gigabit Network Connection PciRoot(0x3)/Pci(0x3,0x4)/Pci(0x0,0x0)/MAC(18c04d711026,1)/IPv6([::]:<->[::]:,0,0)..BO
Boot0005* UEFI: PXE IP6 Intel(R) I350 Gigabit Network Connection PciRoot(0x3)/Pci(0x3,0x4)/Pci(0x0,0x1)/MAC(18c04d711027,1)/IPv6([::]:<->[::]:,0,0)..BO
Boot0006* UEFI OS HD(1,GPT,86ad9d07-0596-4077-965b-bded3e4da2e5,0x800,0x100000)/File(\EFI\BOOT\BOOTX64.EFI)..BO
Boot0000* centos HD(1,GPT,86ad9d07-0596-4077-965b-bded3e4da2e5,0x800,0x100000)/File(\boot\efi\EFI\centos\shimx64.efi)

I wonder if there rhel 8 efibootmgr is compatible with v5.4 ubuntu kernel, w.r.t. nvme drives.

So my efibootmgr issue is the same as

Bug-Ubuntu: https://bugs.launchpad.net/bugs/1891718
Bug: https://github.com/rhboot/efivar/issues/157
Origin: upstream, https://github.com/rhboot/efivar/commit/4e12f997f8b6af76ef65e7045c232b7d642a1af4

because that is not fixed in the CentOS efivar package.

If I set "nvme_core.multipath=N" in /MAAS/r/settings/configuration/kernel-parameters

I can deploy CentOS 8 successfully and ssh into it.

CentOS 7 does not appear to support NVME drives. Deploying onto sda drive works.
CentOS 8 does not appear to support multipath NVME in their efibootmgr.
Deploying with multipath-nvme disabled works, and deploying onto sda drive works.

centos efibootmgr entry is present, working, and boots in all of those cases. And is updated to the new one each time with the new GPT partuuid.

I'm not sure at this point what is wrong, or how to reproduce this.

Somehow I did observe that I had to reboot the machine by-hand, before centos user appeared and was usable.

Can you please explain in more detail what you are observing? What are steps to reproduce this issue? and/or logs?

I wonder if i'm not catching this at the right time.

I can produce this using a couple of the machines in our CI and KVM. The KVM uses /usr/share/OVMF/OVMF_CODE.secboot.fd and all virtio devices[1]. The attached Curtin log is deploying the latest CentOS 8 image to KVM running UEFI. You can see the UEFI entry is successfully created and verified.

Running command ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmp41l7sf8c/target', 'efibootmgr', '--create', '--write-signature', '--label', 'centos', '--disk', '/dev/vda', '--part', '1', '--loader', '/boot/efi/EFI/centos/shimx64.efi'] with allowed return codes [0] (capture=True)

Running command ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmp41l7sf8c/target', 'efibootmgr', '-o', '0003,0007,0004,0001,0002,0006,0000,0005'] with allowed return codes [0] (capture=False)
BootCurrent: 0003
Timeout: 0 seconds
BootOrder: 0003,0007,0004,0001,0002,0006,0000,0005
Boot0000* UiApp
Boot0001* UEFI Misc Device
Boot0002* UEFI Misc Device 2
Boot0003* UEFI PXEv4 (MAC:52540084334D)
Boot0004* UEFI HTTPv4 (MAC:52540084334D)
Boot0005* EFI Internal Shell
Boot0006* UEFI QEMU DVD-ROM QM00001
Boot0007* centos

I manually verified this by SSHing into the VM while it was deploying, preventing rebooting with `sudo touch /tmp/block-reboot`, waiting for MAAS to receive the Curtin log, and then verifying in the Focal ephemeral environment with `sudo efibootmgr -v` that centos was added to the boot order.

# efibootmgr -v
BootCurrent: 0003
Timeout: 0 seconds
BootOrder: 0003,0007,0004,0001,0002,0006,0000,0005
Boot0000* UiApp FvVol(7cb8bdc9-f8eb-4f34-aaea-3ee4af6516a1)/FvFile(462caa21-7614-4503-836e-8ab6f4662331)
Boot0001* UEFI Misc Device PciRoot(0x0)/Pci(0x2,0x4)/Pci(0x0,0x0)N.....YM....R,Y.
Boot0002* UEFI Misc Device 2 PciRoot(0x0)/Pci(0x2,0x5)/Pci(0x0,0x0)N.....YM....R,Y.
Boot0003* UEFI PXEv4 (MAC:52540084334D) PciRoot(0x0)/Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(52540084334d,1)/IPv4(0.0.0.00.0.0.0,0,0)N.....YM....R,Y.
Boot0004* UEFI HTTPv4 (MAC:52540084334D) PciRoot(0x0)/Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(52540084334d,1)/IPv4(0.0.0.00.0.0.0,0,0)/Uri()N.....YM....R,Y.
Boot0005* EFI Internal Shell FvVol(7cb8bdc9-f8eb-4f34-aaea-3ee4af6516a1)/FvFile(7c04a583-9e3e-4f1c-ad65-e05268d0b4d1)
Boot0006* UEFI QEMU DVD-ROM QM00001 PciRoot(0x0)/Pci(0x1f,0x2)/Sata(0,65535,0)N.....YM....R,Y.
Boot0007* centos HD(1,GPT,950e1d23-47b8-4c55-9180-41012f34861b,0x800,0x100000)/File(\boot\efi\EFI\centos\shimx64.efi

Once I rebooted the machine to allow the deployment to finish I checked `efibootmgr -v` in CentOS8 and now I see

# efibootmgr -v
BootCurrent: 0003
Timeout: 0 seconds
BootOrder: 0003,0004,0001,0002,0006,0000,0005
Boot0000* UiApp FvVol(7cb8bdc9-f8eb-4f34-aaea-3ee4af6516a1)/FvFile(462caa21-7614-4503-836e-8ab6f4662331)
Boot0001* UEFI Misc Device PciRoot(0x0)/Pci(0x2,0x4)/Pci(0x0,0x0)N.....YM....R,Y.
Boot0002* UEFI Misc Device 2 PciRoot(0x0)/Pci(0x2,0x5)/Pci(0x0,0x0)N.....YM....R,Y.
Boot0003* UEFI PXEv4 (MAC:52540084334D) PciRoot(0x0)/Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(52540084334d,1)/IPv4(0.0.0.00.0.0.0,0,0)N.....YM....R,Y.
Boot0004* UEFI HTTPv4 (MAC:52540084334D) PciRoot(0x0)/Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(52540084334d,1)/IPv4(0.0.0.00.0.0.0,0,0)/Uri()N.....YM....R,Y.
Boot0005* EFI Internal Shell F...

@ltrager

Thanks for this. I do see that you are able to reproduce this with "normal" non-nvme drivers, and also using KVM.

Let me try to reproduce this again.

Boot0000* centos HD(1,GPT,952730e9-c1fb-4ecb-9e9d-9bb55d99979f,0x800,0x100000)/File(\boot\efi\EFI\centos\shimx64.efi)

$ sudo mount /dev/disk/by-partuuid/952730e9-c1fb-4ecb-9e9d-9bb55d99979f /mnt/

$ ls /mnt/boot/efi/EFI/centos/shimx64.efi
ls: cannot access '/mnt/boot/efi/EFI/centos/shimx64.efi': No such file or directory

"\boot\efi" is the mountpoint of the ESP. But the path that needs to go into the UEFI variable should be the one relative the root of ESP. I.e. "\EFI\centos\shimx64.efi".

Plus, it is probably best to create the entry the same way that the fallback mode would create

cat /mnt/EFI/centos/BOOTX64.CSV
��shimx64.efi,CentOS Linux,,This is the boot entry for CentOS Linux

Aka, "CentOS Linux".

Whatever is creating the centos efi entry is adding a wrong prefix.

suspecting curtin bug

This bug is fixed with commit 56cc9cce to curtin on branch master.
To view that commit see the following URL:
https://git.launchpad.net/curtin/commit/?id=56cc9cce

This bug is believed to be fixed in curtin in version 21.2. If this is still a problem for you, please make a comment and set the state back to New

Thank you.