Devices attached to running instances get reordered

Openstack Focal/Ussuri
Libvirt

When a device (Network or Disk) is attached to a running instance and then the instance is shutoff (via the OS or Nova), the re-render of the xml file reorders the devices. Ubuntu/Linux has the ability to match the network interface to the correct device (when configured properly) but Windows does not. Upon shutdown and start of these instances the instance follows the order of enumeration of the device and the OS then attaches the wrong network configuration to (what it thinks) is the correct interface.

Steps to reproduce:
1) Start an instance
2) Add another Network Interface to that instance while it is running.
3) Shutdown the instance
4) Start the instance again and observe the devices in the instance.

On Windows machines this immediately causes network connection issues as the wrong configuration is being used on the wrong device.

We have not tested this with Nova/VMWare.

Per @krenshaw:

"The PCI slots are being reordered when Nova rebuilds the VM after any sort of hard stop (openstack server stop, evacuate, etc). This causes both the MAC interchange and disk offline issues.

The reason this occurs is that Nova redefines the VM after stop events, up to and including a hard reboot[0]. When this occurs, the VM is regenerated with all currently attached devices, making them sequential within the device type.

This causes reordering when an instance has had volumes and/or networks attached and detached, as devices that are attached after boot are added at the end of the list of PCI slots. On rebuild, these move to PCI slots in sequential order, regardless of the attach/detach order.

Having checked the Nova code, Nova doesn't store PCI information for "regular" non-PCI-passthrough devices. This includes NICs and volumes. Adding this capability would be a feature request with no guarantee of implementation."

We (@setuid @krenshaw) believe it is the metadata that nova is passing to libvirt to re-render the XML file.