Installed image can be missing necessary boot file

This problem occurs when attempting to deploy Ubuntu 18.04, too.

It appears that the ESP's \efi\boot\bootx64.efi is Shim. There's also a copy of fbx64.efi in this directory, but it looks like this Shim isn't configured to look for it, so in the absence of grubx64.efi in this directory, the boot process hangs when Shim is launched. At least, that's my hypothesis.

Machines managed by MAAS are always configured to boot off of the network. When an image is deployed MAAS sends grub, which was loaded over the network, a configuration file[1] which searches for the local boot loader to chain boot to. As per 13.3.1.3 of the UEFI spec[2] MAAS first tries the default location for a local boot loader, \EFI\BOOT\BOOTX64.EFI. If that is not found or fails to load a number of known vendor directories are searched and attempted. If everything fails GRUB exists so the firmware tries the next boot device.

If I keep a close eye on the UEFI machine booting I can see that \EFI\BOOT\BOOTX64.EFI is being loaded but fails to chain load \EFI\BOOT\GRUBX64.EFI. On UEFI QEMU as well as our CI machines GRUB continues, finds /EFI/ubuntu/shimx64.efi and booting succeeds.

Do you have secure boot enabled? If so can you try deploying with it disabled? It may be causing the boot process to lock when BOOTX64.EFI fails to load.

I'm adding GRUB and shim as \EFI\BOOT\BOOTX64.EFI should chain load to the locally installed GRUB. I'm also not sure why GRUB isn't trying the other alternatives its given.

[1] https://git.launchpad.net/maas/tree/src/provisioningserver/templates/uefi/config.local.amd64.template
[2] https://uefi.org/sites/default/files/resources/UEFI_Spec_2_8_final.pdf

Secure Boot is NOT enabled on the affected machine.

So far, I've seen this on only this one server. Other machines, including others deployed from the same MAAS server, deploy and boot fine.

I've discovered that adjusting some firmware options caused the system to deploy normally. Specifically, I adjusted the settings of some PCI devices to favor EFI-mode vs. BIOS-mode device firmware. The warnings noted in my initial bug report still appear on the screen for a few seconds, but the computer now moves past them.

I still think that the fallback (EFI\BOOT\bootx64.efi) boot path should either work (which it does not without a matching EFI\BOOT\grubx64.efi binary) or it should be completely absent. Thus, I believe this is still a bug, albeit one that's being worked around by my modified firmware settings.

To elaborate on the preceding, the options in question are under Advanced -> PCIe/PCI/PnP Configuration in the firmware setup screen. I toggled several from "Legacy" to "EFI." I'm attaching a screen shot. The firmware is an AMI Aptio variety.

I get exactly same issue with HPE DL380 gen10, I can't find right UEFI settings to work around it.
After 2nd reboot the ESP is not there.
If I choose to boot from disk it will boot fine.
Annoyingly, if I make any change in UEFI and leave it in PXE boot, it will boot fine.
Very frustrating. I think I tried 20 different UEFI configs, but still can't get it to work around this issue.
I can change boot order to disk, but then I will lose the maas control over it.

I was not using latest maas available,
After updating from 2.4.2 to 2.6.2 the issue is no longer present

Note that the original bug report was under MAAS 2.6.1, which I believe was the latest non-development release at the time.

Given the comments, it sounds like we need to know more details on what's going on on the node itself. Do we have a system where we can reproduce this on?

I got same issue on Lenovo SR665 server for MAAS 2.8.2 when deploy Ubuntu 20.04. The server UEFI secure boot was disabled.

I didn't find it do the try to boot/load BOOTX64.EFI from ipmiconsole output and if 'cp ubuntu/* BOOT', then above error would disappear.

Below is efi partions and theirs files which deployment by MAAS.
/boot/efi/
/boot/efi/EFI
/boot/efi/EFI/ubuntu
/boot/efi/EFI/ubuntu/grubx64.efi
/boot/efi/EFI/ubuntu/shimx64.efi
/boot/efi/EFI/ubuntu/mmx64.efi
/boot/efi/EFI/ubuntu/BOOTX64.CSV
/boot/efi/EFI/ubuntu/grub.cfg
/boot/efi/EFI/BOOT
/boot/efi/EFI/BOOT/BOOTX64.EFI
/boot/efi/EFI/BOOT/fbx64.efi

Could you please attach the rsyslog log from maas for that machine when this is happening?

The original bug report includes the entire /var/log/maas directory tree, including the rsyslog file for the failing node, meitner.

That said, I've just tried reproducing the problem, and have failed. Meitner is now deploying fine, even when I reconfigure the node's firmware options, as described in comments #5 and #6. I suspect something may have changed in Shim; or perhaps it was something quirky in the machine's boot order that has since changed. (Unfortunately, I didn't record detailed boot order information for the node.)

Zhanglei, can you reproduce this with MAAS 3.0?