Port detach fails when compute host is unreachable

If nova would allow an admin to `force` an unbind, but still queue all the standard cleanup in nova, would that solve this? Is that unreasonable?

The detach-interface is an RPC cast (async) to the nova-compute. When the nova-compute is down the RCP message waits in the queue to be processed. So when the compute comes up it will get the message and detach the interface successfully (I've tested in devstack). In theory nova-api could detect if the compute is down and can unbind the port in neutron (and maybe also de-allocate the resources in placement) if the compute is down. Then nova-api would still cast to the compute to make sure when the compute is up, it can detach the vif from the server.

I agree with gibi, it sounds like we could do something similar to "local delete" (when nova-compute is down and delete is requested, we deallocate ports, volumes, and placement resources) in nova-api for the detach-interface API. There might be some resistance to the idea as "local delete" has been a popular source of bugs in the past. But as always, it's a tradeoff, and if detach-interface while nova-compute is down is a greater pain point, it might be worth adding a "local detach" ability.

I think the only concern here will be the case where nova-compute is down and comes back up, but somehow the RPC message is lost and it never detaches the vif but the port is free to be bound to a new server. Would that be a problem or would it work OK because the port was unbound (and not cause two servers to potentially respond to the same IP, if the port gets bound to a new server)?

Yeah I think it'd work fine with neutron understanding where the port should be bound correctly.

it depnes on how the comptue node is down and if the queue still exists wheater it would get cleand up eventualy. until https://bugs.launchpad.net/oslo.messaging/+bug/1661510 is fix i dont think we can assume that it will definetly get recived and acted on when the compute node comes back up.

on master at least if we detach the port on master the network info cache should get updated with the correct value due to the recent force heal change and that should mean if the vm was stoop and then started it would be started without the port.

to add support for local delete we would need a perodic or a starup task that force referesh the info cache form neutron and checks every vm and determing if the xml should be modified to match the info cache interface built form neutorn. it can be dont but i dont think we can rely on the rpc to do it.

given a workaround to the quota issue would be to delete the server and or loadbalancer and recreate in this case
i would normally set this to low. given the implication that that would be problematic in an octavia context ill mark it as medium but it would be good if you could add context as to why that is not recommended for octavia related instances. i would have assumed you could jsut delete the port and remove it form the load blancer and then re add the port somewhere else.

Sean, I'm not sure I can follow your last comments.

The issue we have is when someone powers down the host with an instances on it, the port detach doesn't complete until the host is brought back up.
This "locks" both quota, but also the fixed IP address on the failed instance. It blocks us from being able to create a new port on a new instance using that quota and/or fixed IP address.
Deleting and detach both get "stuck" on the instance until the host comes back, which it may never do.
This is why this is a high severity issue for users like Octavia.

Abandoned after re-enabling the Octavia launchpad.