Orange Box

Bug #1367482
Comment #20

Comment 20 for bug 1367482

Revision history for this message

Ryan Harper (raharper) wrote on 2014-12-16:

#20

I spent quite a bit of time to see if there was something to do. The current state of pxe control in QEMU isn't ideal. You can disable loading of the option rom which prevents any nic of that *type* from pxe booting. However, there is no control over a per-nic basis.

The other effect of loading a pxe rom, is that *all* nics of a type (virtio, e1000, rtl8139, etc) will be tried. They will be tried in the order they appear on the PCI bus. The order is static.

From what I understand, the unpredictable nature doesn't stem from a QEMU/libvirt side directly, as I said, QEMU will pxe attempt exactly the same every single time. Rather the relatively short time the netboot runs is the cause of the variability as seen by MAAS PXE/DHCP. The first nic issues a bootp/dhcp broadcast request and after a few seconds with no response, it will try the next nic. In the above scenario then it's likely that the first nic times out before it can get a response from MAAS and QEMU switches to the next nic.
The performance/timeout of the bridge reacting to a new device (when KVM launches it adds the guest nic to the software bridge, it has a timeout before packets get forwarded to the devices on the bridge) could be tuned to reduce/delay this, but the problem could still show up on a heavily loaded MAAS when PXE/DHCP is slow to respond.

I've not looked at adjusting the PXE rom timeouts; there are no command line tunables, so it would likely involve generating custom PXE roms (or updating the default, and large default timeouts punish everyone).

For Orange box specifically, I suggest making the following changes to the KVM guest VM to handle this issue:

http://paste.ubuntu.com/9349582/

The changes included switch the first nic to use virtio (instead of rtl8139, unrelated but a better, faster choice), and then switch the second nic to be e1000, and then include the <rom file=''> directive which disables pxe booting on all e1000 nics. The VM now will pxe boot only from the virtio nic. If the bridge is slow or maas is busy, then the VM may not successfully pxe boot. This may or may not be more desirable from an Orange Box perspective.

I spent quite a bit of time to see if there was something to do.  The current state of pxe control in QEMU isn't ideal.  You can disable loading of the option rom which prevents any nic of that *type* from pxe booting.  However, there is no control over a per-nic basis.

The other effect of loading a pxe rom, is that *all* nics of a type (virtio, e1000, rtl8139, etc) will be tried.  They will be tried in the order they appear on the PCI bus.  The order is static.

From what I understand, the unpredictable nature doesn't stem from a QEMU/libvirt side directly, as I said, QEMU will pxe attempt exactly the same every single time.  Rather the relatively short time the netboot runs is the cause of the variability as seen by MAAS PXE/DHCP.  The first nic issues a bootp/dhcp broadcast request  and after a few seconds with no response, it will try the next nic.  In the above scenario then it's likely that the first nic times out before it can get a response from MAAS and QEMU switches to the next nic.
The performance/timeout of the bridge reacting to a new device (when KVM launches it adds the guest nic to the software bridge, it has a timeout before packets get forwarded to the devices on the bridge) could be tuned to reduce/delay this, but the problem could still show up on a heavily loaded MAAS when PXE/DHCP is slow to respond.

For Orange box specifically, I suggest making the following changes to the KVM guest VM to handle this issue:

http://paste.ubuntu.com/9349582/

The changes included switch the first nic to use virtio (instead of rtl8139, unrelated but a better, faster choice), and then switch the second nic to be e1000, and then include the <rom file=''> directive which disables pxe booting on all e1000 nics.  The VM now will pxe boot only from the virtio nic.  If the bridge is slow or maas is busy, then the VM may not successfully pxe boot.  This may or may not be more desirable from an Orange Box perspective.