neutron

you can end up in a state where qvo* interfaces aren't owned by ovs which results in a dangling connection

Bug #1815762 reported by Ian Kumlien
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Won't Fix
Undecided
Unassigned
neutron
Confirmed
Medium
Unassigned

Bug Description

While upgrading to rocky, we ended up with a broken openvswitch infrastructure and moved back to the old openvswitch.

We ended up with new machines working, old machines didn't and it took a while to realize that we had qvo* interfaces that not only wasn't plugged but also wasn't owned by ovs-system - basically the virtual equivalent of forgetting to plug in the cable ;)

This was quickly addressed by running this bash-ism on all nodes:
for x in `ip a |grep qvo |grep @qvb |grep -v ovs-system | awk '{ print $2 '}` ; do y=${x%%"@"*} && ip link delete $y ; done ; docker restart nova_compute

However, nova could pretty easily sanity check this =)

Revision history for this message
sean mooney (sean-k-mooney) wrote :

this might be somthing that could be added to the exsiting
neutron-ovs-cleanup script
that is generated by this entry point https://github.com/openstack/neutron/blob/master/setup.cfg#L49
and impmeneted here https://github.com/openstack/neutron/blob/master/neutron/cmd/ovs_cleanup.py

but this should not live in nova.

Changed in nova:
status: New → Won't Fix
Revision history for this message
Ian Kumlien (pomac) wrote :

Is this script run on each startup? The issue is that nothing will work until it's been run.

Miguel Lavalle (minsel)
Changed in neutron:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Hello Ian:

Can you provide which was the original version?

Did you move from iptables hybrid driver to OVS firewall? Or did you just remove the hybrid driver? In this case, there is a section describing how to handle this upgrade: https://github.com/openstack/neutron/blob/master/doc/source/contributor/internals/openvswitch_firewall.rst#upgrade-path-from-iptables-hybrid-driver

Regards.

Revision history for this message
Ian Kumlien (pomac) wrote :

No, we upgraded to a version of openvswitch that didn't work, it kept busylooping and didn't actually handle any packets - so we reverted, but connections was left in this disconnected state.

We have had this happening several times since then as well - IMHO nova should check the interfaces to see that they are *connected* on startup.

Revision history for this message
Ian Kumlien (pomac) wrote :

(note, we use the fact that nova will set them up if they are missing - but it will not do a small sanity check)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.