maas/pxe cannot boot with nvme device (ovmf works)

Bug #1873662 reported by Guilherme G. Piccoli
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
seabios (Ubuntu)
Confirmed
Low
Unassigned

Bug Description

MAAS relies on PXE to control its nodes (be it a bare-metal host or virtual machine); if machine is ready to get deployed (state ready), MAAS will PXE-boot the machine/guest to distro installer. After the installation, MAAS still PXE-boot the guest, but instructs it to sanboot local disks.

This bug report is related to PXE-booting a qemu guest to sanboot *with NVMe device* (I've tested the emulated one, I can try a PCI-PT too). It fails on seaboot, but works in OVMF. There's no PXE command to boot nvme (or at least I didn't find), but what OVMF seems to do is to fail PXE boot and fallback to local disks, be it SCSI/Virtio-blk or NVMe, and it has the desired effect (MAAS succeeds the deployment).
On seabios, the PXE-boot-to-local fails and the guest is stuck On older seabios (1.10.x), for some reason a PXE boot happens to the commission kernel (a MAAS thing) instead of getting the guest stuck.

The FW logs (and VNC screenshots) are attached - the versions I've used are:

# cat /etc/os-release |grep -i pretty
PRETTY_NAME="Ubuntu 18.04.4 LTS"

# dpkg -l | grep "qemu-kvm\|seabios\|ovmf\|libvirt0\|ipxe-qemu " | tr -s \ | cut -f1-3 -d \
ii ipxe-qemu 1.0.0+git-20190109.133f4c4-0ubuntu3
ii libvirt0:amd64 5.4.0-0ubuntu5.2~cloud0
ii ovmf 0~20191122.bd85bf54-2ubuntu3
ii qemu-kvm 1:4.0+dfsg-0ubuntu9.4~cloud0
ii seabios 1.13.0-1ubuntu1
(I'm using ipxe, seabios and ovmf from Focal, in order to get tests with more updated versions).

Next steps would be to test PCI-PT NVMe boot and upstream qemu+seabios+ipxe on Focal, to determine if the issue is present upstream.

Tags: seg
Revision history for this message
Guilherme G. Piccoli (gpiccoli) wrote :

This could well be a MAAS bug (or even iPXE), as I'm not really sure in which component the flaw is. But given that MAAS+OVMF works, and I've confirmed both seabios and OVMF are using the same ipxe ROM, I believe this is a seabios issue.

MAAS PXE message sent to deployed guests is like below:

$ curl tftp://localhost/ipxe.cfg-52:54:00:ac:e9:7e
#!ipxe
# Boot the locally attached hard drives. On failure try the next locally
# attached hard drive. Fail on the last hard drive, if booting fails.
sanboot --no-describe --drive 0x80 ||
sanboot --no-describe --drive 0x81 ||
sanboot --no-describe --drive 0x82 ||
sanboot --no-describe --drive 0x83

This command was executed inside MAAS VM (the MAC address is for the nvme guest).

Revision history for this message
Guilherme G. Piccoli (gpiccoli) wrote :

This is the VNC screenshot of a failing boot with seabios 1.10 - it eventually booted the commission kernel via iPXE. If using 1.13, the screen is quite similar, but it gets stuck on "Booting from SAN device 0x81".

Revision history for this message
Guilherme G. Piccoli (gpiccoli) wrote :
Revision history for this message
Guilherme G. Piccoli (gpiccoli) wrote :
Revision history for this message
Guilherme G. Piccoli (gpiccoli) wrote :

After building Focal OVMF with (verbose) debug enabled, got this log.

Changed in seabios (Ubuntu):
importance: Medium → Low
Changed in seabios (Ubuntu):
assignee: Guilherme G. Piccoli (gpiccoli) → nobody
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.