powering off and on an instance can result in instance boot failure due to serial port handling race

The following is specific to the libvirt driver.

When we call power_off() it calls _destroy(), which in turn calls self._get_serial_ports_from_guest() and loops over all the serial ports calling serial_console.release_port() on each. This removes the host TCP port from ALLOCATED_PORTS (which is the set of allocated ports on the host).

Then when we call power_on(), it again calls _destroy(), which again calls self._get_serial_ports_from_guest(). This will return the same set of ports that it did before. This is a problem, because those ports could have been allocated to another instance in the meantime!

So in the case where one or more of those ports had been allocated to another instance, we call serial_console.release_port() on them, and remove them from ALLOCATED_PORTS.

Then as part of power_on() we will create new XML with new serial ports, which could select the ports that we just removed from ALLOCATED_PORTS (which are actually in use by another instance). When qemu tries to bind to this port it will fail, causing the instance to error out and stay in the SHUTOFF state.

One possible solution would be to call guest.detach_device() on the "serial" and "console" devices from the guest in the power_off() routine. That way when we call _destroy() in the power_on() routine there wouldn't be any devices returned by _get_serial_ports_from_guest(). This is a bit messy though, so if anyone has any better ideas I'd like to hear about it.