qvo ports are not removed correctly when an instance is deleted immediately after creation

Anatolii, the command you provided doesn't wait until a VM is spawned. Could you please try the same with the --poll option?

That doesn't matter if command waits for instance booting completion or not.
Stale ports should not remain regardless of the way the instance is booted and cleaned up.

Eugene, let's wait until Anatolii answers, ok?

Using --poll options helps to eliminate the issue, so it can be used as a workaround when we launch a bunch of machines "manually". However, as I mentioned before, the customer is heavily using Heat stacks to launch and delete instances, so we need to fix the root cause of this issue because slowing down heat stack (adding WaitConditions?) is not appropriate.

Small update on the issue. I managed to reproduce it in my virtual lab using subsequent calls to boot and delete. It turned out that deletion of interface itself is happening when nova receives the 'network-vif-deleted' event from neutron, but if the instance doesn't exist by that moment, nova fails to delete the interface. The neutron-openvswitch-agent plugin does not create or delete interfaces of the instance itself, but it manages interface's properties like VLANs. If the instance does not exist (what happens after deletion) it cannot bind the created interface to any meaningful VLAN and set it to DEAD_VLAN (4095). This is expected behavior. What we see if we delete an instance right after the request of its creation IMO is predictable. Neutron cannot get in time sending the network-vif-deleted event after a port destruction, and nova drops that event. Not sure if this is fixable at all, therefore I've asked Vladylav Drok to kindly take a look on this issue.

So after doing some investigation I think the following happens:

1. boot request comes in, instance build greenthread is spawned, neutron ports are allocated, info put into instance.info_cache.network_info (asynchronously)
2. instance is deleted, greenthread to handle deletion starts, task_state set to deleting, greenthread is suspended
3. instance build greenthread gets to execute, it sees that task_state became 'deleting' instead of the expected one ('spawning'). during such update of the task_state field an exception is thrown, which triggers deallocate_for_instance method of NetworkAPI
4. this method deletes neutron ports that were created and cleans up cached network info
5. deletion greenthread gets to execute. it goes all the way down to the libvirt driver's cleanup method, which tries to remove the compute nics by using the cached network info, which is now empty. nics are not removed

I'll take a closer look if there is a way to fix this properly, but it is not obvious to me right now :) Does the fact that these devices (qbr, qvb, qvo) are not deleted cause any trouble?

> Does the fact that these devices (qbr, qvb, qvo) are not deleted cause any trouble?
Vladyslav, no.

The main "trouble" is OVS agent complaining about unknown ports, which makes neutron red in monitoring dashboards. OVS agent doesn't detect non-ovs artifacts (qbr, qvb, tap) so it doesn't matter.

Still it would be great to have this issue fixed.

This is not a High priority. This is Low priority. No data is corrupted. No data plane lack of connectivity. No control plane lack of connectivity. The only thing that is a minor issue is some OVS virtual bridges left on the compute host which don't hurt anything.

We don't fix Low impact bugs in MUs.

sla2 for 9.0-updates

sla1 for 9.0-updates

This bug becomes critical when DPDK is enabled, because of hardcoded vhost socket limit in DPDK driver:
http://dpdk.org/browse/dpdk/tree/lib/librte_vhost/socket.c#n94
This limit cannot be increased via configuration, therefore making it essential that there is no buildup of stale interfaces on computes.

Also this bug follows to generate ~1GB logs per hour.

I am able to reproduce this issue in a DPDK-enabled lab with a simple loop:

while true; do nova boot --flavor m1.tiny --image TestVM --nic net-id=59691fd6-a122-4e5a-90a0-8285721a18d4 test1 | tee | grep '^| id ' | awk '{print $4}' | tee | xargs nova delete; done

It took a minute or two to generate 12 stale dpdk socket ports / OVS ports with tag 4095. So the probability is very high and potential impact is high if DPDK is used - just a few hours of running some active VM creation & deletion can bring OpenStack down, i.e. user won't be able to spawn any new VMs.

We need to work further to find a solution, I'll investigate myself but any help is highly appreciated.

I also confirm the root cause to be identical to what has been reported earlier by Denis and Vladyslav - network_info is empty during cleanup, therefore no ports are deleted.

Setting as Won't Fix since there is no solution without refactoring API.