GRUB does not bring up networking when loaded over HTTP

I modified MAAS to only provide GRUB and skip the Shim. In that case HTTP boot still doesn't work. It looks like when GRUB is loaded over HTTP it does not bring up networking. I can still manually load networking with net_dhcp and specify the remote grub.cfg file. GRUB does download the kernel and initrd the remote grub.cfg file specifies but it looks like it never actually excutes them.

"I modified MAAS to only provide GRUB and skip the Shim" => that will not work in secureboot without canonical CA certificates provisioned. If only MS certificate is provisioned (default for most x86 hw) then one must use shim too.

Need additional information as per https://bugs.launchpad.net/maas/+bug/1865515/comments/16

Please provide remote artifacts:
All bootloader files are pulled from the Bionic archive and provided on images.maas.io by lp:maas-images. bootloaders.yaml[1] describes what files are pulled from what packages. You can download the tars at[2]

Please provide local artifacts:
The local artifacts are from the deploy OS. So for Ubuntu its whatever shim/grub is in the archive for that version of Ubuntu. Same goes for CentOS, VMware, Windows, etc.

Keep in mind once HTTP boot is set the remote GRUB drops to the command line before loading the remote grub.cfg. I can only precede with the manual steps detailed above.

Please provide reproducer steps:
1. Install MAAS
2. Configure an UEFI virtual machine to use with MAAS. I would suggest manually creating a UEFI VM with libvirt and adding it to MAAS as a machine.
3. Commission the VM and enable SSH so you can enable HTTP boot
4. SSH into the VM and put HTTP boot before TFTP as described above.
5. Shutdown the machine.
6. Try to deploy any operating system

Please provide details how local artifacts were installed:
Local artifacts are installed by Curtin which gets them from the Ubuntu archive when installing Ubuntu or CentOS archive when installing CentOS.

Please provide list of certs trusted by the node's firmware:
Due to LP:1865515 secure boot was disabled to produce this bug.

[1] https://git.launchpad.net/maas-images/tree/conf/bootloaders.yaml
[2] https://images.maas.io/ephemeral-v3/daily/bootloaders/uefi/amd64/

FYI, reproduced this in LXD virtual machines trying to use UEFI HTTPBOOT.
Similar setup, http-only (no https yet) and no secureboot enabled.

Shim and grub are both retrieved properly over http, then dumped into a grub shell without it ever attempting to download grub.cfg over the network.

net_ls_addr is similarly empty here.

So I managed to make it work by using SUSE's grubx64.efi instead.

Steps are basically:
 - Replace bootx64.efi with the OpenSUSE one
 - Write a grub.cfg shim at the root which does "configfile (http,172.17.16.10:5248)/grub/grub.cfg-default-amd64"

Reboot and everything works as expected.
OpenSUSE Tumbleweed from which the grub build was taken is also on grub 2.04.

So the SUSE srpm has a lot interesting patches that implement:
 * configuring networking from EFI
   - bootp bootpv6
   - dhcp dhcpv4
   - static, v4 & v6, with/without port numbers
   - http https file access via EFI protocol

I don't believe any of that is upstream, and/or only partially, But i should double check.

I think we want to cherrypick:

# Support HTTP Boot IPv4 and IPv6 (fate#320129)
Patch281: 0002-net-read-bracketed-ipv6-addrs-and-port-numbers.patch
Patch282: 0003-bootp-New-net_bootp6-command.patch
Patch283: 0004-efinet-UEFI-IPv6-PXE-support.patch
Patch284: 0005-grub.texi-Add-net_bootp6-doument.patch
Patch285: 0006-bootp-Add-processing-DHCPACK-packet-from-HTTP-Boot.patch
Patch286: 0007-efinet-Setting-network-from-UEFI-device-path.patch
Patch287: 0008-efinet-Setting-DNS-server-from-UEFI-protocol.patch
# UEFI HTTP and related network protocol support (FATE#320130)
Patch420: 0001-add-support-for-UEFI-network-protocols.patch
Patch421: 0002-AUDIT-0-http-boot-tracker-bug.patch

Not sure if we want a feature to automatically scan/find grub.cfg on remote host by ip/mac/uuid/etc:

# bsc#1166409 - Grub netbooting does not search for grub.cfg files with mac
# address or ip address in filename
Patch700: 0001-normal-Move-common-datetime-functions-out-of-the-nor.patch
Patch701: 0002-kern-Add-X-option-to-printf-functions.patch
Patch702: 0003-normal-main-Search-for-specific-config-files-for-net.patch
Patch703: 0004-datetime-Enable-the-datetime-module-for-the-emu-plat.patch

Also they have interesting fixes to limit screen resolution, to make fonts readable out of the box.

> Not sure if we want a feature to automatically scan/find grub.cfg on remote host by ip/mac/uuid/etc:

Currently MAAS only specifies the path to the bootloader, not the bootloader config. It expects the bootloader will automatically request the config from where the bootloader was downloaded from. Right now that is <server>/grub/grub.cfg. That file attempts to chainload <server>/grub/grub.cfg-${net_default_mac} and if that fails <server>/grub/grub.cfg-default-amd64. MAAS could also respond to <server>/grub/grub.cfg-$uuid as we have that information as well. Having grub do that automatically would remove a request and a level of chainloading.

https://git.launchpad.net/maas/tree/src/provisioningserver/boot/uefi_amd64.py

Status changed to 'Confirmed' because the bug affects multiple users.

"MAAS could also respond to <server>/grub/grub.cfg-$uuid as we have that information as well"

In that sentence which $uuid do you mean? UUID of the machine? Something grub can query from smbios?

I mean the output of /sys/class/dmi/id/product_uuid. However I'm not sure how reliable that is. Since I made that comment we've had multiple users report(LP:1893690) some vendors are handing out duplicate UUIDs. MAAS handles this by ignoring any UUID which has been duplicated.

nevermind, uuid is the one from dhcp client-id obviously as implemented / referenced in my earlier comment.

https://launchpad.net/ubuntu/+source/grub2/2.04-1ubuntu40/+build/21017796/+files/grub-efi-amd64-bin_2.04-1ubuntu40_amd64.deb

from this, you can extract /usr/lib/grub/x86_64-efi/monolithic/grubnetx64.efi and drop it into

/var/snap/maas/common/maas/boot-resources/current/bootloader/uefi/amd64/grubx64.efi

and check that all the fixes are good for your deployment too.

For me, I can do httpboot, have dhcp come up by default, and the grub.cfg is loaded.

Hi folks,

It looks like the issue is still around - I just got hit by this with MAAS 3.1.0 on Focal. However, replacing grubx64.efi with the one from /usr/lib/grub/x86_64-efi-signed/grubx64.efi.signed made the trick and I was able to boot without any manual intervention.

Can we consider replacing the grub binary with the one which comes with our distro?

Test if the updated bootloader is free of the issue. Check if Jammy bootloader is backwards-compatible with i386 and if it could be used instead of the Focal one.

Not sure if this bug is still valid as I can't reproduce it anymore on our master branch (using lxd+qemu). HTTP boot over IPv4 seems to work fine - see attachment.