unbalanced UEFI TPL manipulations in iPXE with DOWNLOAD_PROTO_HTTPS enabled

Please add

  -debugcon file:debug.log -global isa-debugcon.iobase=0x402

to the QEMU cmdline, and attach "debug.log". Thanks.

Thanks for the quick response, I'm attaching the debug.log file.

Status changed to 'Confirmed' because the bug affects multiple users.

Vladislav,

The OVMF debug log ends like this (with UEFI protocol GUIDs decoded as
their textual identifiers in edk2):

> [Security] 3rd party image[6D19D18] can be loaded after EndOfDxe: PciRoot(0x0)/Pci(0x3,0x0)/Offset(0x16400,0x4B1FF).
> InstallProtocolInterface: [EfiLoadedImageProtocol] 6D187C0
> Loading driver at 0x00006B1F000 EntryPoint=0x00006B25497 82540em.efi
> InstallProtocolInterface: [EfiLoadedImageDevicePathProtocol] 6D18498
> ProtectUefiImageCommon - 0x6D187C0
> - 0x0000000006B1F000 - 0x00000000000B6E60
> InstallProtocolInterface: [EfiDriverBindingProtocol] 6B50C00
> InstallProtocolInterface: [EfiComponentName2Protocol] 6B50BD0
> ASSERT /home/jenkins/workspace/edk2/rpms/build/edk2-g6ff7c838d0/MdeModulePkg/Core/Dxe/Image/Image.c(1676): Image->Tpl == gEfiCurrentTpl

This final log snippet confirms that a UEFI device driver called
"82540em.efi" is being loaded and started from the option ROM BAR of the
PCI device that is at slot 3, function 0, of the root bridge.

When this UEFI device driver is started, it trips an assert in the
platform firmware. Namely, in the CoreStartImage() function in the
"MdeModulePkg/Core/Dxe/Image/Image.c" source file of edk2:

  //
  // Image has completed. Verify the tpl is the same
  //
  ASSERT (Image->Tpl == gEfiCurrentTpl);

This suggests that the "82540em.efi" driver exits its entry point
function after having raised, but not having restored, the TPL (Task
Priority Level). In other words, the symptom indicates a bug in the UEFI
driver.

I *suspect* (but am not sure) that you are using an e1000 emulated NIC,
and the "82540em.efi" driver exposed in its oprom comes from the iPXE
project:

src/drivers/net/intel.c: PCI_ROM ( 0x8086, 0x100e, "82540em", "82540EM", 0 ),

Therefore I suspect a bug in the iPXE version that the Ubuntu 20.04
upgrade brought to you.

(I can see a number of TPL-related patches in the iPXE git history,
around Feb-Mar 2018. And QEMU loads the iPXE oprom into the emulated
NICs ROM BAR.)

Please try installing different versions of the iPXE package on your
Ubuntu host, and re-run your test (without changing any other elements
of your setup).

Thanks.

(From the UEFI executable name "82540em.efi" in the log, I initially suspected an assigned physical NIC with a buggy flashed-on oprom. But grepping the iPXE tree for "82540em" yields a match, and QEMU loads the iPXE oproms by default into the emulated NICs' ROM BARs.)

Hi Laszlo,
thanks for investigating the problem so rapidly.

So, I downgraded the ipxe-qemu package from 1.0.0+git-20190109.133f4c4-0ubuntu3 (focal) to 1.0.0+git-20180124.fbe8c52d-0ubuntu2 (bionic) and the problem completely disappeared. Your theory looks absolutely correct to me.

For what it's worth, I just discovered that, even with the buggy ipxe-qemu in Focal, the OVMF distributed with QEMU itself (/usr/share/qemu/OVMF.fd) worked, but ONLY with it. I tried with multiple other versions of OVMF and all of them caused QEMU to stuck at boot, probably because of that ASSERT in the 82540em.efi driver.

Vlad

Hi Vlad,

the ipxe-qemu package in Ubuntu (1.0.0+git-20190109.133f4c4-0ubuntu3) is
built with DOWNLOAD_PROTO_HTTPS enabled (in "src/config/general.h").
According to the Ubuntu changelog, this is a new feature added in
"1.0.0+git-20190109.133f4c4-0ubuntu1".

With DOWNLOAD_PROTO_HTTPS enabled, I can reproduce the issue locally,
with iPXE built from source at git commit 133f4c4 (which you report the
issue for), and also at current iPXE master (9ee70fb95bc2).

The issue does not reproduce (with DOWNLOAD_PROTO_HTTPS enabled) at
commit fbe8c52d. This suggests the problem should be bisectable.

If I disable DOWNLOAD_PROTO_HTTPS, then the problem goes away even at
133f4c4 (i.e., the issue is masked).

I've used current edk2 master to test with (14c7ed8b51f6).

Viewed at 133f4c4:

The DOWNLOAD_PROTO_HTTPS feature test macro seems to result in iPXE
attempting to gather entropy. (Likely for setting up TLS connections.)
For entropy gathering, iPXE seems to use an EFI timer, and to measure
jitter across one timer tick. In this, iPXE plays some tricks with the
UEFI TPL (Task Priority Level).

In general, iPXE seems to want to run at TPL_CALLBACK most of the time,
to mask the timer interrupt in most code locations, and drops down to
TPL_APPLICATION only when it actively wants a timer callback (for the
jitter collection, see above).

When the iPXE driver is launched, the StartImage() UEFI boot service
takes a note of the current TPL. It is TPL_APPLICATION (value 4). Then
iPXE seems to perform the above trickery with TPL_CALLBACK & entropy
collection. Finally, after installing EfiDriverBindingProtocol and
EfiComponentName2Protocol, the iPXE driver exits (as expected from a
UEFI driver model driver -- the entry point function is only supposed to
perform some setup steps & install some protocol interfaces). At this
point, StartImage() verifies whether the TPL has been restored to the
same as it was before launching the driver.

Unfortunately, something about the TPL manipulations in iPXE is
unbalanced, because I see the following TPL changes:

- raise: APPLICATION (4) -> CALLBACK (8)
- raise: CALLBACK (8) -> NOTIFY (16)
- raise: NOTIFY (16) -> NOTIFY (16)
- restore: NOTIFY (16) -> NOTIFY (16)
- restore: NOTIFY (16) -> CALLBACK (8)

Note that the final "restore: CALLBACK (8) -> APPLICATION (4)"
transition is missing, before iPXE exits. This is what StartImage()
catches and reports with the failed ASSERT().

So, as I mentioned, the problem is bisectable. Here's the bisection log:

> git bisect start
> # bad: [9ee70fb95bc266885ff88be228b044a2bb226eeb] [efi] Attempt to
> # connect our driver directly if ConnectController fails
> git bisect bad 9ee70fb95bc266885ff88be228b044a2bb226eeb
> # bad: [133f4c47baef6002b2ccb4904a035cda2303c6e5] [build] Handle
> # R_X86_64_PLT32 from binutils 2.31
> git bisect bad 133f4c47baef6002b2ccb4904a035cda2303c6e5
> # good: [fbe8c52d0d9cdb3d6f5fe8be8edab54618becc1f] [ena] Fix spurious
> # uninitialised variable warning on older versions of gcc
> git bisect good fbe8c52d0d9cdb3d6f5fe8be8edab54618becc1f
> # bad: [bc85368cdd311fe68ffcf251e7e8e90c14f8a9dc] [librm] Ensure that
> # inline code symbols are unique
> git bisect bad bc85368cdd31...

Thanks for the whole investigation, Laszlo.
I sent an e-mail to <email address hidden> forwarding your analysis with
a quick summary of mine on the top, indicating the probable first bad commit.

Vlad

Awesome debugging Lazlo and also a really well doen bug report Vladislav - thanks!

As Ubutnu background DOWNLOAD_PROTO_HTTPS is enabled since quite a long time in Ubuntu since bug 1025239.

I don't know if there are better ways nowadays that might allow to disable it in future versions of Ubuntu, but for the already released versions I can't just disable it anyway.
Therefore @Vladislav if the thread you started at <email address hidden> comes to a conclusion/patch please let us know here so that I can consider backporting whatever the final solution will be.

I was happy to contribute, Christian :-)

I just wanted to add that after sending the e-mail to <email address hidden>, I received an automatic response explaining that my e-mail will have to be approved by a moderator because I'm not a member of that mailing list. I just hope that my e-mail won't rot there indefinitely.

Vlad,

you could subscribe to ipxe-devel at <https://lists.ipxe.org/mailman/listinfo/ipxe-devel>, wait until it's confirmed (I think it's automatic, so no moderator approval is needed for subscribing), and then resend your message. You can even stay subscribed -- if you don't want to get the ipxe-devel list traffic, you can click "unsubscribe or edit options", and then "disable mail delivery" (which is *different* from unsubscribing). That way you'd remain a subscriber (so you could post at any time), but you wouldn't get the general list traffic.

hth
Laszlo

Christian,

you can enable DOWNLOAD_PROTO_HTTPS in the traditional BIOS image built from iPXE, and disable it in the UEFI driver built from iPXE. You can still combine both drivers into a combined option ROM. For SeaBIOS guests, there's not going to be any change.

For UEFI guests, see my comment#7 -- you don't need the iPXE HTTPS implementation, because OVMF already contains the edk2 HTTPS BOOT feature (assuming you build OVMF with "-D NETWORK_HTTP_BOOT_ENABLE -D NETWORK_TLS_ENABLE"). The whole point of CONFIG=qemu is to strip the UEFI build of iPXE to a mere Simple Network Protocol driver. Such a build provides only the lowest level hardware abstraction for UEFI, and the rest (IPv[46], TCP/UDP, DHCP, PXE, TFTP, HTTP(S)) are provided on top by the edk2 implementations of various standard UEFI interfaces.

If you want to run a full-featured iPXE build on a UEFI machine (including: in an OVMF guest), you still can, of course; lots of people do that, for good reasons. But that use case is best served by the *standalone UEFI application* build of iPXE (produced at "bin-x86_64-efi/ipxe.efi" by "make"). The UEFI *driver* build of iPXE should be as minimal as possible, in comparison -- just provide SNP for the desired NIC models.

@Lazlo thanks a lot for that awesome experience and guidance!

Config is a bit odd in this package anyway from too many people touching it with different mindset and a lot of history.

There is this from upstream source: src/config/general.h
Which is patched to enable DOWNLOAD_PROTO_HTTPS via debian/patches/enable-https.patch

But there also is $ cat debian/config/general.h:
#define ROM_BANNER_TIMEOUT 0
#define NET_PROTO_IPV6
#define DOWNLOAD_PROTO_NFS
This goes into ./config/local/general.h at the override_dh_auto_configure stage.

I think I should merge with the latest version from Debian, then see how I can configure HTTPS with some but not the other builds and then run everything through a bigger regression check.
I'll start a merge of the latest version in bug 1884758 and then check out disabling DOWNLOAD_PROTO_HTTPS for EFI.

The recent edk2 builds have -DNETWORK_HTTP_BOOT_ENABLE=TRUE -DNETWORK_TLS_ENABLE
Recent as in 2020.05-2 which means >=groovy.

For the eventual SRU to Focal things are more complex as there -DNETWORK_TLS_ENABLE was missing.

The above was an FYI, but is should be fine as outlined by Lazlo this isn't needed as since ipxe 1.0.0+git-20180124.fbe8c52d-0ubuntu4 we use CONFIG=qemu and in comment #7 he explained that in this case "totally don't need (or even *use*) the iPXE HTTPS infrastructure (the entropy gathering that trips the ASSERT seems spurious to me, with CONFIG=qemu). With CONFIG=qemu, iPXE provides the UEFI SNP (Simple Network Protocol) interface on top of the e1000 NIC, and the crypto stuff (if any) is done by the platform firmware (edk2 / OVMF)"

Note: the bisection result of d8c500b7945e ("[efi] Drop to TPL_APPLICATION when gathering entropy" is in <=133f4c4 but > fbe8c52d which means for Ubuntu releases that would be affected >=Disco.

@Lazlo - are combined roms breaking your suggestion to "just disable https in efi roms"?
In the build for the efi roms it uses this at some point:
  src/util/catrom.pl src/bin-i386-pcbios/82540em.rom src/bin-x86_64-efi/82540em.efirom > src/bin-combined/82540em.efirom

So the *efi* file in ipxe-qemu will be a combined rom (since 2013).
Of these the legacy one will have HTTPS enabled and the EFI part won't.
Would you expect that this is a problem for the suggested workaround (I don't but would be glad if you would let me know what you think about it)?

Christian,

what you describe seems to be exactly what I propose. Namely:
- build "82540em.rom" with HTTPS enabled,
- build "82540em.efirom" with CONFIG=qemu, and HTTPS disabled,
- create a combined option ROM image from the above two, using "catrom.pl".

Thanks
Laszlo

Thank you Lazlo, going forward that is the process that I have it execute then.
On the SRU I'll "only" disable HTTPS on EFI roms and we can take a look if nothing else stops working but this case here would be fixed.

--

One question thou - asking for help to be able to make this an SRU at soem point.

Vladislav initially reported the test could just be "qemu -bios ovmf".
I tried this in various Ubuntu releases like:
  qemu-system-x86_64 -bios /usr/share/ovmf/OVMF.fd
But without any other setup that properly gives the edk tianocore logo followed by some PXE tries and failing to boot. After the timeouts all passed it eventually gives me the UEFI shell.

I'd have hoped this behaves differently in Bionic (working) and Focal (failing) per the initial report - but it seems to be the same.

My point is that I fail to recreate the bug in the existing releases, which is a requirement (to be able to verify the fix) for the SRU.

@Vladislav (or @Lazlo since you said you can reproduce it) - is there now a better way known how to trigger this.

I used

qemu-system-x86_64 \
  -enable-kvm \
  -monitor stdio \
  -drive if=pflash,snapshot=on,format=raw,file=OVMF.fd \
  -global e1000.romfile=82540em.combined.rom \
  -debugcon file:debug.log \
  -global isa-debugcon.iobase=0x402

where "OVMF.fd" was built from edk2 at then-master (14c7ed8b51f6 -- see comment 7), and "82540em.combined.rom" is the combined oprom (with HTTPS enabled in the EFI driver too) built from the problematic iPXE version(s).

OVMF was built with "-b DEBUG". ("-b RELEASE" would remove the ASSERT()s from the firmware modules.)

"debug.log" captures the firmware debug output. That's the file that ends with the ASSERT failure seen in comment 4.

Following [1] I was building my test OVMF as guided by Lazlo.
That way I was able to use the combined e1000 EFI of the Ubuntu packages vs the debug OVMF build.

Using that I can confirm the behavior (Bionic working, Focal failing).

$ qemu-system-x86_64 -enable-kvm -monitor stdio -drive if=pflash,snapshot=on,format=raw,file=/root/edk2/Build/OvmfX64/DEBUG_GCC5/FV/OVMF.fd -global e1000.romfile=/usr/lib/ipxe/qemu/efi-e1000.rom -debugcon file:debug.log -global isa-debugcon.iobase=0x402

#/usr/lib/ipxe/qemu/efi-e1000.rom is from package ipxe-qemu
#/root/edk2/Build/OvmfX64/DEBUG_GCC5/FV/OVMF.fd is the debug build, I'll attach that file here to eas repo in other places

Symptoms when failing:
- UI never leaves the "initializing" state
- in the debug.log the bad case asserts with
  ASSERT /root/edk2/MdeModulePkg/Core/Dxe/Image/Image.c(1676): Image->Tpl == gEfiCurrentTpl

That it seems to work fine with the ovmf build that is in Focal 0~20191122.bd85bf54-2ubuntu3 makes the SRU of this less urgent IMHO. And also resolves some of my remaining confusion since I've known that EFI boots in general work - and it seems (at least in this test POV) - that only a newer or different built OVMF triggers the issue.

[1]: https://wiki.ubuntu.com/UEFI/EDK2#Set_up_build_environment

I beg your pardon Lazlo, but one more question on CONFIG=qemu.
Since it was introduced config=QEMU was exported for efi and legacy roms.
But it seems for legacy roms like 82540em.rom CONFIG=qemu being set or not doesn't make a difference.

I I just look at src/config/qemu/* vs src/config/ there is a massive difference and I see it showing up in the build log, so the export has "some" effect.

Grepping for qemu.*82540em I see

Old:
/tmp/buildlog_ubuntu-focal-amd64.ipxe_1.0.0+git-20190109.133f4c4-0ubuntu3_BUILDING.txt:35050:gcc -DARCH=i386 -DPLATFORM=pcbios -DSECUREBOOT=0 -march=i386 -fomit-frame-pointer -fstrength-reduce -falign-jumps=1 -falign-loops=1 -falign-functions=1 -mpreferred-stack-boundary=2 -mregparm=3 -mrtd -freg-struct-return -m32 -fshort-wchar -Ui386 -Ulinux -DNVALGRIND -Iinclude -I. -Iarch/x86/include -Iarch/i386/include -Iarch/i386/include/pcbios -Os -g -ffreestanding -Wall -W -Wformat-nonliteral -fno-stack-protector -fno-dwarf2-cfi-asm -fno-exceptions -fno-unwind-tables -fno-asynchronous-unwind-tables -Wno-address -Wno-stringop-truncation -fno-PIE -no-pie -ffunction-sections -fdata-sections -include include/compiler.h -DASM_TCHAR='@' -DASM_TCHAR_OPS='@' -DCONFIG=qemu -DLOCAL_CONFIG=qemu -DOBJECT=version -DBUILD_NAME="\"82540em.rom\"" \
/tmp/buildlog_ubuntu-focal-amd64.ipxe_1.0.0+git-20190109.133f4c4-0ubuntu3_BUILDING.txt:47807:gcc -DARCH=x86_64 -DPLATFORM=efi -DSECUREBOOT=0 -fstrength-reduce -fomit-frame-pointer -falign-jumps=1 -falign-loops=1 -falign-functions=1 -m64 -mno-mmx -mno-sse -fshort-wchar -Ui386 -Ulinux -DNVALGRIND -fpie -mno-red-zone -Iinclude -I. -Iarch/x86/include -Iarch/x86_64/include -Iarch/x86_64/include/efi -Os -g -ffreestanding -Wall -W -Wformat-nonliteral -fno-stack-protector -fno-dwarf2-cfi-asm -fno-exceptions -fno-unwind-tables -fno-asynchronous-unwind-tables -Wno-address -Wno-stringop-truncation -ffunction-sections -fdata-sections -include include/compiler.h -DASM_TCHAR='@' -DASM_TCHAR_OPS='@' -DCONFIG=qemu -DLOCAL_CONFIG=qemu -DOBJECT=version -DBUILD_NAME="\"82540em.efidrv\"" \

New:
ipxe_1.0.0+git-20190125.36a4c85-5ubuntu1~ppa1_amd64.build:43490:gcc -DARCH=x86_64 -DPLATFORM=efi -DSECUREBOOT=0 -fstrength-reduce -fomit-frame-pointer -falign-jumps=1 -falign-loops=1 -falign-functions=1 -m64 -mno-mmx -mno-sse -fshort-wchar -Ui386 -Ulinux -DNVALGRIND -fpie -mno-red-zone -Iinclude -I. -Iarch/x86/include -Iarch/x86_64/include -Iarch/x86_64/include/efi -Os -g -ffreestanding -Wall -W -Wformat-nonliteral -fno-stack-protector -fno-dwarf2-cfi-asm -fno-exceptions -fno-unwind-tables -fno-asynchronous-unwind-tables -Wno-address -Wno-stringop-truncation -ffunction-sections -fdata-sections -include include/compiler.h -DASM_TCHAR='@' -DASM_TCHAR_OPS='@' -DCONFIG=qemu -DLOCAL_CONFIG=qemu -DOBJECT=version -DBUILD_NAME="\"82540em.efidrv\"" \

But at least on the output size of 82540em.rom I see no difference.
Hence my question - does CONFIG=qemu have no effect on the legacy roms?
If it has an effect what is the recommended config for them - setting it or not?
(Right now I have no more set CONFIG=qemu for legacy roms, but since it would be a change I want to be sure it really is a no-op or at least what is r...

Christian,

> But it seems for legacy roms like 82540em.rom CONFIG=qemu being set or not doesn't make a
> difference.

(1) That's my understanding as well; from the following original iPXE commits anyway:

- a15c0d7e868a ("[efi] Allow user experience to be downgraded", 2015-07-22),
- a200ad462e69 ("[build] Add named configuration for qemu", 2015-07-22).

(I played a part in those commits existing.)

(2) However, just to be 100% sure, I recommend *not* changing the CONFIG=<whatever> setting for the traditional BIOS build. If you have had CONFIG=qemu there, then keep it. If you haven't, then don't add it now.

What matters is that the EFI build be performed with CONFIG=qemu.

(FWIW, last time I looked, in RHEL downstream we used to build iPXE with CONFIG=qemu covering *both* the traditional BIOS image and the EFI image. But, I don't think that's enough reason for you to diverge now from your previous BIOS image settings in Ubuntu. I don't think you should risk regressions with that. Really, what matters is that the EFI image be built with CONFIG=qemu.)

Thanks
Laszlo

Thanks Lazlo, I'll keep the legacy roms as CONFIG=qemu then.
I didn't plan to change this on an SRU anyway, but going forward I wanted to adapt this to be correct. Hearing that in RH you also used CONFIG=qemu covering *both* is kind of re-assuring to keep it like that for now.

This bug was fixed in the package ipxe - 1.0.0+git-20190125.36a4c85-5ubuntu1

---------------
ipxe (1.0.0+git-20190125.36a4c85-5ubuntu1) groovy; urgency=medium

  * Merge with Debian unstable (LP: #1884758). Remaining changes:
    - Split grub integration from ipxe->grub-ipxe.
      - d/control: add package and adapt dependencies
      - d/[grub-]ipxe.install: move some files to grub-ipxe
      - rename d/ipxe.post* to d/grub-ipxe-post*
      [updated to match 1.0.0+git-20190125.36a4c85-5]
    - d/util/check-rom-sizes, d/rules: check sizes of generated roms to avoid
      accidentally breaking KVM live migration on updates/fixes.
      [updated for new and dropped rom file names]
    - debian/copyright: update copyright information to satisfy lintian
      dep5 checks (LP: 1747071)
    - Build ROMs for QEMU with CONFIG=qemu (LP: 1789319)
      [updated to match 1.0.0+git-20190125.36a4c85-5 and be more verbose]
    - debian/patches/handle-dhcp-nack.patch: Handle DHCP NAK and send a
      re-discover. (LP 1707999)
    - d/p/0005-strip-802.1Q-VLAN-0-priority-tags.patch: Strip 802.1Q VLAN 0
      priority tags; Fixes PXE when VLAN tag is 0. (LP: 1805920)
    - d/tree/ipxe/etc/grub.d/20_ipxe: Make grub-ipxe work under UEFI
      (LP: 1811496)
      - Use ipxe.efi under UEFI
      - Save default entry when iPXE is selected
    - d/tree/ipxe/etc/grub.d/20_ipxe: Identify ipxe grub menu entry in
      an easier way (LP: 1858374)
  * Dropped changes
    - new upstream release 20190109.133f4c4 [superseded by new upstream]
    - new upstream release 20180124.fbe8c52d [superseded by new upstream]
    - Revert to the former git snapshot 20150424.a25a16d
      [superseded by new upstream]
      - This brings back debian/patches/0002-Don-t-use-libiberty.patch as needed
        on the older source.
      - Adapt d/p/0001-rom-change-banner-timeout.diff.patch to former state to
        match old source.
      - drop d/p/util-elf2efi-GNU_SOURCE.patch as it was not needed on old
        source
    - Fix FTBFS with newer perl versions [upstream]
    - d/p/0004-fix_no-pie_option.patch: correct -no-pie option
      to build without pie
      [ still carried before ]
    - Install ne.rom as pxe-ne2k_isa.rom
      - d/ipxe-qemu.install: Install ne.rom as pxe-ne2k_isa.rom.
      - d/ipxe-qemu.links: compat link for ne.rom
      [ no more needed, was an old xen hvmloader bug ]
    - d/ipxe-qemu.links: Add compat links from /usr/share/qemu
      to /usr/lib/ipxe/qemu.
      [ no more needed, transition from trusty ]
    - add new rom for vmxnet3 (LP 1737211) [in Debian now]
    - Add e1000e firmware for qemu. (closes 884240) [in Debian now]
    - d/control: ipxe-qemu breaks qemu-system-x86 <2.11 [no more needed]
    - d/p/enable-https.patch: Enable HTTPS support.
      [resolved per rom type in d/rules now]
  * Added changes
    - d/rules: only enable https on non EFI roms. This lets EFI handle https
      itself and avoids breakage in TPL manipulations (LP: #1882671)
    - d/util/check-rom-sizes: fix if size does exactly match

ipxe (1.0.0+git-20190125.36a4c85-5) unstable; urgency=medium

  * Cleanup src/bin correctly. (closes: #952275)

ipxe (1.0.0+git-20190125.36a4c85...

I have prepared a merge proposals and PPA test builds for Focal/Eoan
E-MP => https://code.launchpad.net/~paelzer/ubuntu/+source/ipxe/+git/ipxe/+merge/386647
E-PPA => https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4126/+packages
F-MP => https://code.launchpad.net/~paelzer/ubuntu/+source/ipxe/+git/ipxe/+merge/386648
F-PPA => https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4127/+packages

For Eoan/Focal we need to be sure that the OVMF builds from edk2 can really take over the HTTPS functionality. Because edk2 itself for Debian/Ubuntu only got enabled later in >=Groovy:

  edk2 (2020.05-2) unstable; urgency=medium
      * Enable https boot support, thanks to Dimitri John Ledkov. LP: #1883114.

This comes down to:
-COMMON_FLAGS = -DNETWORK_HTTP_BOOT_ENABLE=TRUE -DSECURE_BOOT_ENABLE=TRUE
+COMMON_FLAGS = -DNETWORK_HTTP_BOOT_ENABLE=TRUE -DNETWORK_TLS_ENABLE -DSECURE_BOOT_ENABLE=TRUE

Therefore once we drop HTTPS from the ipxe-qemu combined efi roms expecting that OVMF will take over this we need to ensure this can work without above enabling being available in Eoan/Focal as well.

/me looks for a good way to verify that as I'm unsure if the test mentioned in bug 1883114 will really proved what we need in regard to dropping https here. Maybe an actual OVMF boot via HTTPS should be set up. If there are suggestions for a good way to test that this OVMF-HTTPS-takeover works as expected I'm open to them.

If it turns out that we need to enable it in edk2/ovmf before we can go on in ipxe/ipxe-qemu we we can upload ipxe-qemu with a versioned BREAKS to the older ovmf package (to avoid https is dropped in 'ipxe-qemu', but not yet enabled in the 'ovmf'). But if needed backporting bug 1883114 becomes a pre-req to SRU this bug here.

We will need quite some time to ensure this isn't breaking things.
The merge proposal was reviewed and I'll upload to Focal-unapproved now.
The intention is to not have me testing in advance and then having a short time in -proposed. Instead I think for this case it will be helpful to have it in proposed early, but longer to increase the amount of 3rd-party/people testing it.

Therefore I have added block-proposed tags right away.

Hello Vladislav, or anyone else affected,

Accepted ipxe into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ipxe/1.0.0+git-20190109.133f4c4-0ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Test #1:

 * Test the attached OVMF that triggers the bug:
qemu-system-x86_64 -enable-kvm -monitor stdio -drive if=pflash,snapshot=on,format=raw,file=OVMF-14c7ed8b51f6-DEBUG-enabled.fd -global e1000.romfile=/usr/lib/ipxe/qemu/efi-e1000.rom -debugcon file:debug.log -global isa-debugcon.iobase=0x402

Focal as-is with ipxe-qemu 1.0.0+git-20190109.133f4c4-0ubuntu3 fails:

ASSERT /root/edk2/MdeModulePkg/Core/Dxe/Image/Image.c(1676): Image->Tpl == gEfiCurrentTpl

Upgrading to proposed:
root@f-ipxe:~# apt install ipxe-qemu=1.0.0+git-20190109.133f4c4-0ubuntu3.1
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  ipxe-qemu
1 upgraded, 0 newly installed, 0 to remove and 28 not upgraded.
Need to get 903 kB of archives.
After this operation, 1749 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 ipxe-qemu all 1.0.0+git-20190109.133f4c4-0ubuntu3.1 [903 kB]
Fetched 903 kB in 0s (1963 kB/s)
(Reading database ... 47652 files and directories currently installed.)
Preparing to unpack .../ipxe-qemu_1.0.0+git-20190109.133f4c4-0ubuntu3.1_all.deb ...
Unpacking ipxe-qemu (1.0.0+git-20190109.133f4c4-0ubuntu3.1) over (1.0.0+git-20190109.133f4c4-0ubuntu3) ...
Setting up ipxe-qemu (1.0.0+git-20190109.133f4c4-0ubuntu3.1) ...

With the version from proposed the crash/assert no more happens.

Test #2
 * We pad the rom sizes to be sure, but never the less double check
   migrations between Bionic <-> Focal (known to break on size changes)

Manual size check (can be seen in build log):
OK: efi-e1000e.rom is exactly 524288 bytes as expected
...

Seems ok, a regression test doing cross release migrations with proposed enabled is running

Test #3:

 * Extra Test: HTTPS boot a uEFI guest with the efi roms from ipxe-qemu
   with old/new ipxe-qmeu code. This shall ensure that OVMF can really take
   over as-is or if we need bug 1883114 before we can do so.
   Details TBD when I'm doing these tests

I created a q35 guest in libvirt without a disk and set it to run in uEFI mode via OVMF.
Starting that without further setup runs into an EFI loader that can't find anything to boot.

Start PXE over IPv4
...
Not Found
Start HTTP Boot over IPv4
...
Not Found
-> into interactive boot-failed menu

As I mentioned before in comment #26 Focals EDK2 didn't have HTTPS enabled yet, only in Groovy.

Therefure using the OVMF of groovy and the ipxe-qemu package from Focal-proposed I set up a test.

$ cp ovmf-groovy/usr/share/OVMF/OVMF_VARS.fd test-vars.fd
$ qemu-system-x86_64 -enable-kvm -drive if=pflash,format=raw,readonly,file=/home/ubuntu/ovmf-groovy/usr/share/OVMF/OVMF_CODE.fd -drive if=pflash,format=raw,file=test-vars.fd -monitor stdio

We can see that in this OVMF build the OVMF device manager has the option to enroll TLScerts. But TBH I haven't ever used this setup to then HTTPS boot through EFI/OVMF.

I found [1] but before going through all the lengths to set this up I wonder for further regression testing I wonder if there at all was a way to get HTTPS boot working in EFI mode with:
a) https enabled /usr/lib/ipxe/qemu/efi-e1000e.rom
b) not https enabled /usr/share/OVMF/OVMF_CODE.fd

I'm a bit lost in all the rom/boot/https/loader options.
I beg your pardon but @Lazlo do you know if above mentioned way existed and might - now that we take https away from (a) - be regressing?
If so which way would this need to be set up to be tested?
Is [1][2] a proper way to exercise this in Focal "using the https in e1000e" or would that only work with the HTTPS enabled OVMF of groovy?

[1]: https://en.opensuse.org/UEFI_HTTPBoot_Server_Setup
[2]: https://edk2-docs.gitbook.io/getting-started-with-uefi-https-boot-on-edk-ii/introduction

P.S. cross release migration tests still running

The TPL manipulation issue in iPXE is fixed as of commit http://github.com/ipxe/ipxe/commit/2ae5d4338

Building an iPXE ROM with HTTPS enabled will now initialise with no problems in qemu.

Michael

Hello Christian,

For *some* form of UEFI HTTPS boot, you have to enable *at least one* of
the {edk2, iPXE} HTTPS stacks. I'm unfamiliar with the Ubuntu releases,
but my understanding is the following:

Ubuntu release edk2 HTTPS enabled iPXE HTTPS enabled iPXE TPL regression
-------------- ------------------ ------------------ -------------------
Bionic no <don't know> no
Focal no yes yes
Groovy yes (bug 1883114) no (this bug) masked (this bug)

In Groovy, you can work around the iPXE TPL regression by disabling the
iPXE HTTPS stack (i.e., in the efi-e1000e option ROM). Because, you can
effectively "replace" it with the edk2 HTTPS stack in the platform
firmware (in the OVMF binary), per bug 1883114.

In Focal, if you do the same to iPXE, you can't fall back to the edk2
HTTPS stack in OVMF -- because bug 1883114 is out of scope for Focal,
AIUI.

However, disabling the iPXE HTTPS stack in Focal would not cause a
regression, in my opinion. That's because in Focal you can't boot the
"OVMF + efi-e1000e" combination *at all* -- you don't get far enough in
the boot process to even *attempt* HTTPS boot (or a boot from another
kind of media, for that matter).

Thus in Focal, no form of *UEFI boot* (HTTPS or otherwise) has ever
worked, so there's nothing to regress by disabling the iPXE HTTPS stack
in "efi-e1000e.rom".

Sorry about the malformed table in comment 33 -- that's not my doing. I laid it out correctly; Launchpad messed it up by squeezing whitespace. Here it is again, using dots rather than spaces.

Ubuntu.release..edk2.HTTPS.enabled..iPXE.HTTPS.enabled..iPXE.TPL.regression
--------------..------------------..------------------..-------------------
Bionic..........no..................<don't.know>........no.................
Focal...........no..................yes.................yes................
Groovy..........yes.(bug.1883114)...no.(this.bug).......masked.(this.bug)..

Regression tests completed, no issues migrating in/out of updates nor between releases due to changing sizes (That mostly covers the non EFI roms thou).

I want to also do some more manual tests with EFI guests in that regard.

Hi Michael, thank you for the fix, and for mentioning it here. I didn't ignore your comment 32 when I was writing what would become comments 33 and 34 -- I think we must have been writing our comments in parallel, and I simply didn't see yours.

Christian, now you should be able to resolve this LP ticket without modifying anything on the packaging side, by backporting upstream iPXE commit 2ae5d4338.

Thanks!

  <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
  <nvram template='/usr/share/OVMF/OVMF_VARS.fd'>/var/lib/uvtool/libvirt/images/bionic.VARS.fd</nvram>

makes it run eif ovmf EFI like
-drive file=/usr/share/OVMF/OVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/uvtool/libvirt/images/bionic.VARS.fd,if=pflash,format=raw,unit=1

Still mgriates well from Bionic -> Focal and Back
B
$ virsh migrate --unsafe --live bionic qemu+ssh://10.225.185.44/system
F
$ virsh migrate --unsafe --live bionic qemu+ssh://10.225.185.35/system
B
$ virsh console bionic
Connected to domain bionic
Escape character is ^]
Ubuntu 18.04.4 LTS bionic ttyS0
bionic login:

So that also works well.
The only possible option missing to be tested is the real https boot via Focals OVMF (which didn't have https enabled), but unless someone comes back explaining that there is an odd combination that could have got that going we should be good.

Let is sit a bit more time in proposed to be sure and wait for feedback.
Then we can declare it fully verified.

I saw your update on refresh - yeah despite feeling sort of safe on the change as-is this fix seems even better for an SRU.

Let me get that into groovy (there the packaging change made sense, no need to turn that back).
And from there SRU it to Focal.

Thank you Michael and Lazlo for the work and discussion on this.

fix as URL => https://github.com/ipxe/ipxe/commit/2ae5d4338661b65c63eb5cb1a96e5b803fe7d620

This bug was fixed in the package ipxe - 1.0.0+git-20190125.36a4c85-5ubuntu2

---------------
ipxe (1.0.0+git-20190125.36a4c85-5ubuntu2) groovy; urgency=medium

  * d/p/lp-1882671-efi-Raise-TPL-during-driver-entry-point.patch: fix the
    formerly avoided efi TPL issues (LP: #1882671)

 -- Christian Ehrhardt <email address hidden> Thu, 16 Jul 2020 14:36:54 +0200

Fix uploaded for SRU to focal-unapproved.

Hello Vladislav, or anyone else affected,

Accepted ipxe into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ipxe/1.0.0+git-20190109.133f4c4-0ubuntu3.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Testing the actual case:
$ dpkg -S /usr/lib/ipxe/qemu/efi-e1000.rom
ipxe-qemu: /usr/lib/ipxe/qemu/efi-e1000.rom

root@f-ipxe:~# apt-cache policy ipxe-qemu
ipxe-qemu:
  Installed: 1.0.0+git-20190109.133f4c4-0ubuntu3.2
  Candidate: 1.0.0+git-20190109.133f4c4-0ubuntu3.2
  Version table:
 *** 1.0.0+git-20190109.133f4c4-0ubuntu3.2 500
        500 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1.0.0+git-20190109.133f4c4-0ubuntu3.2 500
        500 http://ppa.launchpad.net/ci-train-ppa-service/4156/ubuntu focal/main amd64 Packages
     1.0.0+git-20190109.133f4c4-0ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

$ qemu-system-x86_64 -enable-kvm -monitor stdio -drive if=pflash,snapshot=on,format=raw,file=OVMF-14c7ed8b51f6-DEBUG-enabled.fd -global e1000.romfile=/usr/lib/ipxe/qemu/efi-e1000.rom -debugcon file:debug.log -global isa-debugcon.iobase=0x402

With the fix this now boots through to the actual EFI prompt and tries to initialize boot options. The log file no more lists the assertion.

Testing the sizes went well, they are int he right size buckets as they were before (no major change by the patch).

Note: We have not touched the HTTPs capability in the SRU which makes this much saver in this try#2. Therefore also the re-testing of these isn't needed this time.

This overall LGTM and was in proposed for an extra-while without negative feedback, setting verification-done.

This bug was fixed in the package ipxe - 1.0.0+git-20190109.133f4c4-0ubuntu3.2

---------------
ipxe (1.0.0+git-20190109.133f4c4-0ubuntu3.2) focal; urgency=medium

  * Revert the changes of the non released version
    1.0.0+git-20190109.133f4c4-0ubuntu3.1 as there is a less impactful
    fix more suited for an SRU.
  * d/p/lp-1882671-efi-Raise-TPL-during-driver-entry-point.patch: fix the
    formerly avoided efi TPL issues (LP: #1882671)

ipxe (1.0.0+git-20190109.133f4c4-0ubuntu3.1) focal; urgency=medium

  * only enable https on non EFI roms. This lets EFI/OVMF handle https
    itself and avoids breakage in TPL manipulations (LP: 1882671)
    - d/p/enable-https.patch: drop old global way to Enable HTTPS support
    - d/rules: enable https on non EFI roms.
    - d/util/check-rom-sizes: fix if size does exactly match

 -- Christian Ehrhardt <email address hidden> Thu, 16 Jul 2020 16:51:30 +0200

The verification of the Stable Release Update for ipxe has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Per comment #32, fixed in upstream iPXE commit 2ae5d4338, setting ticket status for iPXE to "Fix Committed".