Baremetal instance stuck in BUILD state following ironic node tear down or delete

Does seem like [2] should really be raising build failed, like the previous step.

The problem is comparing when you delete and instance via the API (and making sure we correctly keep the instance deleted) vs failing due to something else doing the delete during a build.

Either way, stuck in building state is the worst possible outcome.

Agreed, we definitely want to still allow an asynchronous instance delete during build. I think that is the scenario that InstanceNotFound was meant to cover, but it has been incorrectly overloaded by the ironic virt driver.

Oddly, the main exception raised in the nova ironic virt driver when things go wrong is InstanceDeployFailure, but this is not referenced anywhere outside that driver except for the class definition in exception.py.

Reading through this, this bug looks valid to me. Setting to Medium based on the fact that a user has to delete an ironic node out-of-band from nova while the nova instance is building in order to encounter the bug.

The analysis in the bug report makes sense, that the ironic driver is raising InstanceNotFound when calling ironic after the baremetal node was deleted out-of-band of nova. Nova treats it as "nova instance not found" and thus thinks there's nothing to do with the instance state.

I do wonder if it would be correct for the ironic driver to instead raise InstanceDeployFailure (or another new exception such as IronicNodeNotFound) if an ironic node GET call returns 404. I can't think of a reason the ironic driver should raise InstanceNotFound unless it has deleted the nova instance itself.

This idea is based on looking at how we handle a delete via the nova API while a baremetal instance is building. While the instance is building in the driver spawn method, the _wait_for_active [1] method is looping. If a user requests a delete of the instance, the driver loop during build will see the task_state == DELETING and will raise InstanceDeployFailure as a result.

Then, the compute manager doesn't handle the InstanceDeployFailure exception [2] and will raise the RescheduledException. The RescheduledException will be caught [3] and when retries are exceeded, the networks/volumes will be cleaned up and the instance set to ERROR state.

[1] https://github.com/openstack/nova/blob/6bf11e1dc14afad78b11d980c2544a3dc41579ff/nova/virt/ironic/driver.py#L466-L469
[2] https://github.com/openstack/nova/blob/6bf11e1dc14afad78b11d980c2544a3dc41579ff/nova/compute/manager.py#L2199
[3] https://github.com/openstack/nova/blob/6bf11e1dc14afad78b11d980c2544a3dc41579ff/nova/compute/manager.py#L1932