OVS drops RARP packets by QEMU upon live-migration causes up to 40s ping pause in Rocky

Just want to make sure I understand this correctly after reading the bug you referenced.

1. The instance is live-migrated
2. The neutron-ovs-agent on the target node configures the port, but it's after libvirt has
   already starting sending RARPs out
3. A couple of RARPs make it out if you set a config option high enough, for example, 10

I would have figured neutron would have notified nova, and then it would have completed things, triggering the RARPs to happen after that event, but this is not my area of expertise. I'll ask someone that's more familiar with live migration operations to take a look and give their perspective.

this happens as a result of the fact that os-vif is currently not plugging the ovs port and that is instead being done by libvirt.

i had to revert that change due to other neutron issues which slawek has resolved.
https://review.openstack.org/#/c/636824
this nova change https://review.openstack.org/#/c/602432/ tests delegating the port plugging to os-vif. it depends on this change to the requirements repo https://review.openstack.org/#/c/636139/1
which forces the use of os-vif 1.13.1.

when i reverted https://review.openstack.org/#/c/631829/ it was purly to unblock the gate.
the code was correct however i could not merge https://review.openstack.org/#/c/602432/ due
to the unrelated neutron and cinder gate failures.

i expect that we will merge https://review.openstack.org/#/c/636061/ to os-vif soon.
that will allow callers of os-vif to choose if it should create the ovs interface allowing neutorn to wire up the port before qemu starts and preventing the RARPs from being lost.

it should be noted that this will only happen with kernel ovs and the ovs conntrack security group driver are deployed. for ovs-dpdk or kenel ovs with iptabels os-vif always plugs the interface not libvirt and we do not race on libvirt adding the interface, qemu sending the RARPs and neutron wiring up the interface on the br-int.

I've added the os-vif component based on Sean's comment.

Thanks Sean for detailing the history around this issue. Let us fix the known issue Sean is aware first.

For the record:

(1) At this moment in Rocky, the only deployment scenario I don't see the long ping pause is when VM uses vxLAN and native firewall. Both native OVS and OVS-dpdk are tested, I don't see different behavior between native OVS and OVS-dpdk.

Below are the details observed, consistent and reproducible:

> for VM using vxLAN, long ping delay is only with legacy firewall, and it happens on the direction when the migrating VM goes to a compute that has had its network before. No issue with native firewall.

> for VM using vLAN, long ping delay is with both legacy firewall and native firewall, and it happens on the direction when the migration VM goes to a compute that hasn't had its network before, and the the delay is consistent about 40s.

(2) I am also trying to increase RARP# from 5 to 10 at the QEMU side, and is told the following patch (to be merged, not yet) would allow changing the announce-rounds and the other announce timings at run-time:

http://lists.gnu.org/archive/html/qemu-devel/2019-02/msg01486.html

Please note the qemu series:
  http://lists.gnu.org/archive/html/qemu-devel/2019-02/msg04128.html

which:
  a) Makes these announce timings and repetitions configurable
  b) Allows a set of announces to be triggered on command.

Hopefully they'll get merged soon.

Fix proposed to branch: master
Review: https://review.openstack.org/640258

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.openstack.org/640258
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

so how can we deal with this issue? @sean mooney

im going to unabandon the patch i was working on https://review.openstack.org/640258
and try and get back to this over the next 2 weeks or so.

i had some other issue come up which ment i need to put this on hold for a while.

ill be at the ptg the week after next so ill try to have some updated patches ready to discuss with
people in person and see if we can get an agreed path forward.

Note that the qemu changes mentioned in #5 are merged into qemu v4.0

Fix proposed to branch: master
Review: https://review.opendev.org/661921

Fix proposed to branch: master
Review: https://review.opendev.org/661938

I think there are 2 problems cause connectivity broken in the live-migration
1. When the VM is migrated to the destination, and the VM send the rarp packets, but because it's too fast, the openflow and tag haven't been configed in br-int, then the rarp packets will be drop.

2. When the VM is migrated to the destination, the openflow and tag have been configed, then VM send rarp packet, but table 71 flow will drop these packets, because high priority flow(65 70 80 95) doesn't match the rarp packets, only low priority flow(10) will match, but its action is drop, so the packets still cannot been sent out. The table 71 flows:
 cookie=0x6c5958fed07888a7, duration=428986.816s, table=71, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=110,ct_state=+trk actions=ct_clear,resubmit(,71)
 cookie=0x6c5958fed07888a7, duration=24454.927s, table=71, n_packets=0, n_bytes=0, idle_age=24454, priority=95,icmp6,reg5=0x2d,in_port=45,icmp_type=130 actions=resubmit(,94)
 cookie=0x6c5958fed07888a7, duration=24454.927s, table=71, n_packets=3, n_bytes=210, idle_age=24441, priority=95,icmp6,reg5=0x2d,in_port=45,icmp_type=133 actions=resubmit(,94)
 cookie=0x6c5958fed07888a7, duration=24454.927s, table=71, n_packets=1, n_bytes=78, idle_age=24450, priority=95,icmp6,reg5=0x2d,in_port=45,icmp_type=135 actions=resubmit(,94)
 cookie=0x6c5958fed07888a7, duration=24454.927s, table=71, n_packets=0, n_bytes=0, idle_age=24454, priority=95,icmp6,reg5=0x2d,in_port=45,icmp_type=136 actions=resubmit(,94)
 cookie=0x6c5958fed07888a7, duration=24454.927s, table=71, n_packets=0, n_bytes=0, idle_age=24454, priority=70,icmp6,reg5=0x2d,in_port=45,icmp_type=134 actions=resubmit(,93)
 cookie=0x6c5958fed07888a7, duration=24454.927s, table=71, n_packets=1, n_bytes=42, idle_age=24450, priority=95,arp,reg5=0x2d,in_port=45,dl_src=fa:16:3e:b4:db:09,arp_spa=192.168.100.141 actions=resubmit(,94)
 cookie=0x6c5958fed07888a7, duration=24454.927s, table=71, n_packets=91, n_bytes=8818, idle_age=24446, priority=65,ip,reg5=0x2d,in_port=45,dl_src=fa:16:3e:b4:db:09,nw_src=192.168.100.141 actions=ct(table=72,zone=NXM_NX_REG6[0..15])
 cookie=0x6c5958fed07888a7, duration=24454.927s, table=71, n_packets=2, n_bytes=678, idle_age=24450, priority=80,udp,reg5=0x2d,in_port=45,tp_src=68,tp_dst=67 actions=resubmit(,73)
 cookie=0x6c5958fed07888a7, duration=24454.927s, table=71, n_packets=0, n_bytes=0, idle_age=24454, priority=80,udp6,reg5=0x2d,in_port=45,tp_src=546,tp_dst=547 actions=resubmit(,73)
 cookie=0x6c5958fed07888a7, duration=24454.927s, table=71, n_packets=0, n_bytes=0, idle_age=24454, priority=70,udp,reg5=0x2d,in_port=45,tp_src=67,tp_dst=68 actions=resubmit(,93)
 cookie=0x6c5958fed07888a7, duration=24454.927s, table=71, n_packets=0, n_bytes=0, idle_age=24454, priority=70,udp6,reg5=0x2d,in_port=45,tp_src=547,tp_dst=546 actions=resubmit(,93)
 cookie=0x6c5958fed07888a7, duration=24454.927s, table=71, n_packets=1, n_bytes=90, idle_age=24441, priority=65,ipv6,reg5=0x2d,in_port=45,dl_src=fa:16:3e:b4:db:09,ipv6_src=fe80::f816:3eff:feb4:db09 actions=ct(table=72,zone=NXM_NX_REG6[0..15])
 cookie=0x6c5958fed07888a7, duration=24454.927s, table=71, n_packets=1, n_bytes=90, idle_age=24450, priority=10,reg5=0x...

I use some virsh commands to avoid the first situation, modify the migration speed to 1M/s, make sure the flows and tag are configed in br-int before live-migration completed.
# virsh migrate-setspeed instance-0000003b --bandwidth 1
# virsh migrate-getspeed instance-0000003b
# virsh migrate --live --persistent --undefinesource instance-0000003b qemu+tcp://node-3/system

And add a new rarp flow into table=71:
priority=95,ct_state=-trk,rarp,reg5=0x13,in_port=19,dl_src=fa:16:3e:09:3d:10 actions=resubmit(,94)

Then there is no more packets dropped in several live-migration tests。

Hi, sean

the root cause of this issue seams related to the neutron new port binding api. see: https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/neutron-new-port-binding-api.html

in rocky release, live migration activate the port by using port binding api in post_live_migration process. However, the libvirt start new instances and the new instance become alive before the port comes to active, and so the RARP packets are finally lost.

we are verified this by revert the port binding api logical, and then the problem solved. like below:
in neutron/agent/rpc.py
class CacheBackedPluginApi(PluginApi):

        if (port_obj.device_owner.startswith(
315 constants.DEVICE_OWNER_COMPUTE_PREFIX) and
316 binding[pb_ext.HOST] != host):
317 LOG.debug("Device %s has no active binding in this host",
318 port_obj)
319 return {'device': device,
320 n_const.NO_ACTIVE_BINDING: True}

skip this if branch, so that the port is always in active state.

also we need to skip port binding api used by nova:
in nova/network/neutronv2/api.py
    def supports_port_binding_extension(self, context):
        """This is a simple check to see if the neutron "binding-extended"
        extension exists and is enabled.

        The "binding-extended" extension allows nova to bind a port to multiple
        hosts at the same time, like during live migration.

        :param context: the user request context
        :returns: True if the binding-extended API extension is available,
                  False otherwise
        """
        self._refresh_neutron_extensions_cache(context)
        return constants.PORT_BINDING_EXTENDED in self.extensions

we directly return false for supports_port_binding_extension, so nova will not call port binding api during live migration. The legacy way is used

we confirm that because we manually call the activate port binding api to activate the port on destination during migration before the vms actived on destination, then the problem is also dispeared.

since the neutron port binding api has its own advantages, so how can we solve it thoroughly. Is that possible to activate the port binding before the vm shutting down on the source host and vm being running on the destination host? @sean mooney

no we cant activate teh port biding before the vm is runing on the the dest unless you are using post copy live migration.

if you are using post copy live migrate we recive an event form libvirt when the vm is pasused and we activate the port binding at that point which is the first point where we can safely activate the port.

when we activate teh port on the destiatnion the port binding on the source is deactivated and in theory the network backend should prevent any network traffic form been sent or received on the source node. meaning if the ovs agent was correctly isolating the interface when it was deactivated then you would have a netwrok partation if we just activated it before we migrate which is why we do not do that. we cannot start the vm on the dest before stoping it on the souce node as that is entirely managed out side of nova control by libvirt.

if you are not using post copy we activate teh bining in the same place we woudl ahve previously bound the port in post_livemigration.

you are correct that we still have fallback code for neutron version that dont support multiple port bindign for live migration that takes the old path. if hardcoding the check to false fixes the problem it impleies the neutron_l2_agent is incorrectly wiring up ports on hosts that the port is not bound too which is not correct behaviour.

https://review.opendev.org/#/c/640258 allows neutron to wire up the destination port before the vm is created on the destination. https://review.opendev.org/#/c/602432/ was intended to prevent the race between libvirt and the l2 agent.

that is the only way i know of too fix this currently.

i could add a workaround config option to nova to forcully disable the multiple portbinding flow for live migration but that would really just be a hack and it would increase network downtime in some cases so it could not be enabled by default. it also has other issues like the inablity to migrate between ovs hosts with different firewall driver backends.

Hello,
I did not understand if I must patch only nova/network/neutronv2/api.py and nova/workarounds.py
or also neuttron/agent/rpc.py as yao ning suggested.

Add for #15, we should also skip in neutron/agent/rpc.py for workaround to disable port binding:

diff --git a/neutron/agent/rpc.py b/neutron/agent/rpc.py
index 130b18e..5ee396c 100644
--- a/neutron/agent/rpc.py
+++ b/neutron/agent/rpc.py
@@ -311,13 +311,6 @@ class CacheBackedPluginApi(PluginApi):
         binding = utils.get_port_binding_by_status_and_host(
             port_obj.bindings, constants.ACTIVE, raise_if_not_found=True,
             port_id=port_obj.id)
- if (port_obj.device_owner.startswith(
- constants.DEVICE_OWNER_COMPUTE_PREFIX) and
- binding[pb_ext.HOST] != host):
- LOG.debug("Device %s has no active binding in this host",
- port_obj)
- return {'device': device,
- n_const.NO_ACTIVE_BINDING: True}
         net = self.remote_resource_cache.get_resource_by_id(
             resources.NETWORK, port_obj.network_id)
         net_qos_policy_id = net.qos_policy_id

diff --git a/nova/network/neutronv2/api.py b/nova/network/neutronv2/api.py
index f73fee4..088c3f0 100644
--- a/nova/network/neutronv2/api.py
+++ b/nova/network/neutronv2/api.py
@@ -1226,8 +1226,7 @@ class API(base_api.NetworkAPI):
         :returns: True if the binding-extended API extension is available,
                   False otherwise
         """
- self._refresh_neutron_extensions_cache(context)
- return constants.PORT_BINDING_EXTENDED in self.extensions
+ return False

     def bind_ports_to_host(self, context, instance, host,
                            vnic_type=None, profile=None):

The neutron agent must be patched on compute node ?
And nova on controllers ?

The neutron agent rpc.py on compute nodes contains the code you suggested so it does not need to be patched, right ?

I patched the nova/network/neutronv2/api.py as you suggested and the nova/conf/workarounds.py as suggested by Sean here:

https://review.opendev.org/#/c/724386/

I restarded all nova and neutron services.

Any case it does not solve the problem on provider networks on stein and rocky.
To reproduce the issue I tried with an instance that does not generate traffic (I also disabled its chronyd daemon). After live migration it stops to respond.
If I enable the vm chronyd with a polling time of 2 seconds, it responds after live migration because it sends packets every 2 seconds.

Both are needed.

* nova-conductor and nova-compute services with workaround patch
* neutron/agent/rpc.py neutron-openvswitch-agent service on compute node by removing the lines described in https://bugs.launchpad.net/neutron/+bug/1815989/comments/19

Restart the services and do live migration.

I am sorry . I have not understand the output of diff command .Now is clear. I am going to test it

It works.
I changed the nova/conf/workarounds.py and nova/network/neutronv2/api.py only on controllers nodes.
I changed neutron/agent/rpc,py on on compute nodes (deleting lines as you suggested).

Restarded services and now it works :-)

Thanks

anyone help to fix a workaround in neutron?

These patches seems to work fine on stein but not on rocky.

Interesting having issues with this now after moving to Train and CentOS 8.

I can workaround this by sending the new announce_self (which generates RARP and/or GARP or w/e the NIC driver implements itself) to the instance after it's been migrated to another compute node and stops responding.

virsh qemu-monitor-command instance-000000e6 --hmp announce_self

I wonder if we can workaround this somehow if there is no clear path in sight with the patches that has been proposed already. Since libvirt doesn't support it we cannot set migration parameters to use the new announce parameters.

Note that you need QEMU >= 4.0 to support announce_self

Related fix proposed to branch: master
Review: https://review.opendev.org/741529

Rodolfo do you have any clear path on how we can work this bug?

Reviewed: https://review.opendev.org/640258
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7fd2725cb14b81f442eb57a38755829270ff2c43
Submitter: Zuul
Branch: master

commit 7fd2725cb14b81f442eb57a38755829270ff2c43
Author: Sean Mooney <email address hidden>
Date: Fri Mar 1 04:43:20 2019 +0000

    Do not skip ports with ofport unset or invalid

    This change removes the "_check_ofport" function and its use form
    the ovs_lib.py file.

    By skipping ports without a unique ofport in the "get_vifs_by_ids"
    and "get_vifs_by_id" functions, the OVS agent incorrectly treated
    newly added port with an ofport of -1 as removed ports in the
    "treat_devices_added_or_updated" function.

    Co-Authored-By: Rodolfo Alonso Hernandez <email address hidden>

    Change-Id: I79158baafbb99bee99a1d687039313eb454d3a9b
    Partial-Bug: #1734320
    Partial-Bug: #1815989

Related fix proposed to branch: master
Review: https://review.opendev.org/748296

Change abandoned by sean mooney (<email address hidden>) on branch: master
Review: https://review.opendev.org/748296

Sorry for asking again, do we have a clear path on how to solve this?

Also hoping for guidance on how to address this. Thanks!

we figured out what was cause issue for the neutron openvswitch firewall driver yesterday.
rodolfo has propsoed the testing patch https://review.opendev.org/#/c/753314/ bugfix.
that should allow https://review.opendev.org/#/c/602432/20 to move forward and hopefully merge next week once the neutron patch is merged. the nova patch also requires https://review.opendev.org/#/c/751642/5 which should also be merged shortly.

once these are all merged on the master branch we can start the backport process.

the main code for the fix has been available for some time but there were a number of edgecases that had to be adressed. we started looking at these patches again about 2 weeks ago an i think
the main edgecaes have now been addressed.

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/754475

@sean-k-mooney - Thanks for the quick response. We're using the hybrid driver, but also experience the RARP issues. However, looking back through this bug, it seems like https://review.opendev.org/#/c/640258 might address our issue, especially since the issue appears to be due to the amount of time it takes to apply the VLAN tag to the OVS port.

Do you think we can backport https://review.opendev.org/#/c/640258 all the way back to (at least) Stein?

for the hybrid iptables driver os-vif adds a veth pair between the ovs bridge an a linux bridge so teh of-port will not be -1 so it will have no effect in that case.

this only changes teh behavior when using the ovs firewall driver.

but yes we will proably backport it to queens.

Just to shime in, we also have this issue using the hybrid iptables driver. But if https://review.opendev.org/#/c/640258 solves the issue and is backported we are happy :)

Reviewed: https://review.opendev.org/753314
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c8a819aff4db62e58192af0a272a7f1ce7923146
Submitter: Zuul
Branch: master

commit c8a819aff4db62e58192af0a272a7f1ce7923146
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Sep 24 09:44:47 2020 +0000

    Filter out port with invalid ofport in OVS firewall

    Since [1], "get_vif_port_by_id" is also returning ports with an
    invalid ofport. OVS firewall cannot set an OpenFlow rule for a port
    without a valid ofport. "get_ovs_port" should filter out those ports.

    Related-Bug: #1815989
    Related-Bug: #1734320

    [1]https://review.opendev.org/#/c/640258/

    Change-Id: Id12486b3127ab4ac8ad9ef2b3641da1b79a25a50

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/756478

After reading through Sean's latest message again, I interpret it as the above mentioned patch only fixes OVS firewall related issues and the issue is still there when running the iptables_hybrid setup.

the issue does not happen when using the iptables_hybrid firewall driver.
it only happens when using the the openvswtich driver.

the underlying root cause is a race condition between libvirt and neutron that only
happens when libvirt adds the tap device to ovs. when using the iptables_driver
that does not happen.

so unless there is a seperate issue in the neutron l2 agent the iptabeles driver is unaffected.

This issue happens even without firewall. It happens with openvswitch tap interfaces with native plug and disabled security groups (enable_security_group=false in [securitygroup] config section).

the issue is not related to the firewall used but rather a side effect of which firewall is enabel if any.

sepcific the iptables one set hybrid_plug=true and the rest set it to false.

when hybrid_plug=false libvirt plugs the tap into ovs causeing a race.
this will also happen if you disable security groups

anytime you use hybrid_plug=false the race happens.

most people dont know what hybrid plug is and do not disable security groups so i was just using the firewall driver as a proxy to not need to go in to that level of detail

https://www.rdoproject.org/networking/networking-in-too-much-detail/ describs what hybrid plug is in detial but its basically where we add intermediate linux bridges and veth pairs.

in that config we wait for neutron to finish wireing up the port fully before we start the live migration at the libvirt level and libvirt just add the vm tap to the linux bridge

with hybred_plug=false libvirt adds the tap directly to ovs just before it unpauses the vm and the vm start races with the neturon l2 agent or sdn contoler configuring the ovs bridge.

Do we have a clear path on how to solve this on rocky?

yes but likely not for a few months
im trying to get this merged in master once that is actully done we can start backporting.
at present all the neutron patches we are aware of are now merged there are 2 needed for nova at present once those merge we can start backporting. it has been quite hard to get core review on this feature which is why the fixes have been in flight for 2+ years. i am hopeful this will be resolved before the end fo the year and we will have done the backports but i also said that 2 years ago and i could not get people to actully review the chagnes.

the best way to drive this forward is to test the changes and review them.

@sean-k-mooney

First of all, thank you for the work you've put into this issue and for clarifying the actual cause/effect here.

I don't know that I can speak for everyone using the hybrid driver, but in our testing, we kept an eye on the OVS bridge during live migrations and found that migrations in which the Neutron ports attached to br-int had VLAN tagging applied in < 4 seconds were successful 100% of the time (no drops). If the vlan tag application took > 5 seconds, and especially > 7 seconds, the migration would result in ~5-40 seconds of connectivity loss to the VM.

Although the comment references ovs firewall, driver, https://bugs.launchpad.net/neutron/+bug/1414559/comments/49 calls out the same behavior with:

"When the VM is migrated to the destination, and the VM send the rarp packets, but because it's too fast, the openflow and tag haven't been configed in br-int, then the rarp packets will be drop."

In our case, it would seem the VM is already sending RARPs prior to the VLAN tag being set on the port.

Could it be 'live_migration_wait_for_vif_plug=true' is really what we're after, here?

live_migration_wait_for_vif_plug=true is the default value.
if you have set live_migration_wait_for_vif_plug=false then yes it would result in
packet loss and the mac learning frame not propagating if the vlan tag application took a protracted period of time.

when live_migration_wait_for_vif_plug=true we wait for the network-vif-plugged events to be recived from neutron in pre-livemigrate before we tell libvirt to migrate the guests.
if its set to false we do not wiat for the event and just proceed to start the migration once the rpc call too plug the vifs completes without waiting for neutron to wire up the ports with the vlans and flow rules.

so if you have live_migration_wait_for_vif_plug=false you will see similar issue to the behavior with this bug bug since true is the default you should only see that behavior if you opted in by setting it to false. if you currently have it set to false and you are using ml2/ovs then you shoudl set it to true.

it only exts because some network backend do not reliably send event. ovs is not one fo the backed that is broken in that way so it shoudl always be set to true when using ml2/ovs and ovs in general.

Reviewed: https://review.opendev.org/756478
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=24dd977c2217bb10baf2cf226eab2bd0e5b47d62
Submitter: Zuul
Branch: stable/victoria

commit 24dd977c2217bb10baf2cf226eab2bd0e5b47d62
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Sep 24 09:44:47 2020 +0000

    Filter out port with invalid ofport in OVS firewall

    Since [1], "get_vif_port_by_id" is also returning ports with an
    invalid ofport. OVS firewall cannot set an OpenFlow rule for a port
    without a valid ofport. "get_ovs_port" should filter out those ports.

    Related-Bug: #1815989
    Related-Bug: #1734320

    [1]https://review.opendev.org/#/c/640258/

    Change-Id: Id12486b3127ab4ac8ad9ef2b3641da1b79a25a50
    (cherry picked from commit c8a819aff4db62e58192af0a272a7f1ce7923146)

The race exists for iptables_hybrid based deployments when live_migration_wait_for_vif_plug=true (default value) as well.

The source compute node does the pre live migration and waits for the network-vif-plugged which it receives, the live migration starts, the VM is resumed on destination but the destination compute node binds the port AFTER the instance has already been resumed.

To understand the race issue.

* The pre live migration is complete when source compute node gets network-vif-plugged event

2020-10-23 10:39:31.634 3460854 INFO nova.compute.manager [-] [instance: 9ef8fcee-c1cf-4d2e-8b14-2b43c31044f6] Took 2.83 seconds for pre_live_migration on destination host compute-02.
2020-10-23 10:39:32.200 3460854 DEBUG nova.compute.manager [req-7f2c3034-c0b4-4e6b-9209-280638dcd2e1 6283ca84a2ff4cc099fcfd8e50550910 3a28d0f6b65a44c2aa1bbffbfa8bb2ea - default default] [instance: 9ef8fcee-c1cf-4d2e-8b14-2b43c31044f6] Received event network-vif-plugged-f83f20ad-feff-4369-a752-a81964bcfd52 external_instance_event /usr/lib/python3.6/site-packages/nova/compute/manager.py:9273

* Then the instance is resumed on the destination compute node

2020-10-23 10:39:35.467 2082170 INFO nova.compute.manager [req-5c20ab33-21eb-48b8-950f-85807ebc1559 - - - - -] [instance: 9ef8fcee-c1cf-4d2e-8b14-2b43c31044f6] VM Resumed (Lifecycle Event)

* But the port is not really updated or fixed on the destination compute node until after that

2020-10-23 10:39:37.504 2096718 DEBUG neutron.agent.resource_cache [req-3b8c2e3f-4e62-446b-b9db-f6bf12012ab0 f1fc63f1306549a0b1aba80875aac683 3a28d0f6b65a44c2aa1bbffbfa8bb2ea - - -] Resource Port f83f20ad-feff-4369-a752-a81964bcfd52 updated <a lot of port binding data here>

It also passes another round of fixing the port at

2020-10-23 10:39:37.859 2096718 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-d0d6ab14-b56e-4d91-9030-7f422465f628 - - - - -] Port f83f20ad-feff-4369-a752-a81964bcfd52 updated

and the done line is not until

2020-10-23 10:39:38.572 2096718 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-d0d6ab14-b56e-4d91-9030-7f422465f628 - - - - -] Configuration for devices up ['f83f20ad-feff-4369-a752-a81964bcfd52'] and devices down [] completed.

This means that there is a race of about ~3 seconds there when the instance is resumed vs when the port is bound.

Now the question is, nova is properly waiting for the network-vif-plugged event but that is not really the time when the port is ready, so is there any other event that we should/could wait for or is this a neutron issue in the end?

the binding of the port should not be requried for neutno to have wired up the port so that it has netwrok connectivity. before rocky neutron woudl wireup the port on any host at least for the in tree drivers. after rocky nova prebinds the port by creating a second port bingding. it then activates the port binding in post live migration.

neutron should have made the port ready before sending the network-vif-plugged event.
in the iptabels case the vm tap device is added to an intimidate linux bridge which is not visible to neutron. the neutron l2 agent just sees a one end fo the veth pair that interconnect the linuxbridge to ovs. so nova is expecting that when network-vif-plugged is sent neutorn has done everything required to all networking to work when the qemu instance starts.

so yes this is a neutron issue but the outstanding question is whether neutron is correct to act as it is currently, neutron is tying the link state to the state of the port binding on the destination host. when there was only one port binding it was unspecified what the network backend did. it coudl fully wire up the port on a host without a binding or it could ignore the new port until the binding was updated to that host. ml2 ovs did the former and some other thidpart sdn solution( i think odl) ignored the port as there was no binding on that host.

if the correct behavior is to only allow traffic to flow once the binding is active then neutron is doing the right thing if not then its a seperate neutron bug from the libvirt race.

it should be noted that post-copy live migration with the force-complete live migration timeout action will signifcatly shorten the time to activate teh dest port binding.

if you are not already using post-copy migration you should change to it to mitgate the delay and in general speed up the live migration.

these are the setting i recommend are used by default for all openstack deployment using libvirt

[libvrirt]
live_migration_permit_post_copy=true
live_migration_timeout_action=force_complete
live_migration_permit_auto_converge=true

Thanks Sean for your in-depth information, really appreciate. I've added another neutron bug linking to this one.

https://bugs.launchpad.net/neutron/+bug/1901707

Reviewed: https://review.opendev.org/c/openstack/nova/+/602432
Committed: https://opendev.org/openstack/nova/commit/a62dd42c0dbb6b2ab128e558e127d76962738446
Submitter: "Zuul (22348)"
Branch: master

commit a62dd42c0dbb6b2ab128e558e127d76962738446
Author: Stephen Finucane <email address hidden>
Date: Fri Apr 30 12:51:35 2021 +0100

    libvirt: Delegate OVS plug to os-vif

    os-vif 1.15.0 added the ability to create an OVS port during plugging
    by specifying the 'create_port' attribute in the 'port_profile' field.
    By delegating port creation to os-vif, we can rely on it's 'isolate_vif'
    config option [1] that will temporarily configure the VLAN to 4095
    (0xfff), which is reserved for implementation use [2] and is used by
    neutron to as a dead VLAN [3]. By doing this, we ensure VIFs are plugged
    securely, preventing guests from accessing other tenants' networks
    before the neutron OVS agent can wire up the port.

    This change requires a little dance as part of the live migration flow.
    Since we can't be certain the destination host has a version of os-vif
    that supports this feature, we need to use a sentinel to indicate when
    it does. Typically we would do so with a field in
    'LibvirtLiveMigrateData', such as the 'src_supports_numa_live_migration'
    and 'dst_supports_numa_live_migration' fields used to indicate support
    for NUMA-aware live migration. However, doing this prevents us
    backporting this important fix since o.vo changes are not backportable.
    Instead, we (somewhat evilly) rely on the free-form nature of the
    'VIFMigrateData.profile_json' string field, which stores JSON blobs and
    is included in 'LibvirtLiveMigrateData' via the 'vifs' attribute, to
    transport this sentinel. This is a hack but is necessary to work around
    the lack of a free-form "capabilities" style dict that would allow us do
    backportable fixes to live migration features.

    Note that this change has the knock on effect of modifying the XML
    generated for OVS ports: when hybrid plug is false will now be of type
    'ethernet' rather than 'bridge' as before. This explains the larger than
    expected test damage but should not affect users.

    [1] https://opendev.org/openstack/os-vif/src/tag/2.4.0/vif_plug_ovs/ovs.py#L90-L93
    [2] https://en.wikipedia.org/wiki/IEEE_802.1Q#Frame_format
    [3] https://answers.launchpad.net/neutron/+question/231806

    Change-Id: I11fb5d3ada7f27b39c183157ea73c8b72b4e672e
    Depends-On: Id12486b3127ab4ac8ad9ef2b3641da1b79a25a50
    Closes-Bug: #1734320
    Closes-Bug: #1815989

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/790447

Hello, I got same issue on centos 7 train and I am waiting if there is some workaround.
Ignazio

Hello, as suggested by Rodolfo Alonso I tried to apply the following patches to centos 7 train:
https://review.opendev.org/q/Iee323943ac66e566e5a5e92de1861832e86fc7fc

Unfortunately serveral errors appears in neutron-server.log

Probably all patches are for python3 ?
Ignazio

Ignazio: most probably as current openstack master is tested against py3, see:
https://opendev.org/openstack/neutron/src/branch/master/setup.cfg#L9

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/797142

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/797144

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/797291

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/797316

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/797428

Hello, I got same issue on kolla ansible wallaby with ubuntu source images.
Ignazio

Hello, I did not understand if this bug affects only openvswitch or also ovn.
Please, anyone can explain ?

ovn is broken in a different way but with a similar effect.
there is yet another vector for a race between libvirt and ovn in the ovn case.

ovn will not install openflow rules until libvirt creates the tap on the destinatnion host.
as a result there is an interval of time where the vm can be runnign on the dest but ovn has not installed the flows if the

there was a bug in the nova fix which is resolved by
https://review.opendev.org/c/openstack/nova/+/797142/1
https://review.opendev.org/c/openstack/nova/+/797428

basically the nova fix was turned off due to an issue in the code related to who python properties work.

that is corrected by those last too patch and that is being squshed into the backport of the nova patch.

now that we know ovn has different requirement that we must fullfile to mitigate the race condition in the ovn case i am prototyping https://review.opendev.org/c/openstack/os-vif/+/798055
as a solution taht will actully also fix the ovn varent of this.

funementailly if you nighevly live migrate between two hsts with ovn you will always have a race condition between libvirt and ovn unless we can remove the depency on the of-portid that only gets assigned wehn the device is created on the datapath on the host.
to work around this we are oging to create per port ovs bridges and connnoect those to the br-int

this is similar to how hybrid plug works with ml2/ovs and iptabel or to how we implement trunk port in ml2/ovs. unlike hybrid plugs we will not be using linux bridges and we will not have the performace impact that intoduces

the trunk bridge concept is not new in ml2/ovs so the port bridge design is more or less well tested since we are using the same techniuq just for a different usecase.

ironicly the fix for ovn will also work for ml2/ovs and would have avoid some of the change we already made for ml2/ovs to adress this in the first place but it was intially dismissed as it results in a more complex bridge toplogy which at the time we did nto want to intoduce since ml2/ovs did not need it.

ignazio (cassano) https://review.opendev.org/c/openstack/nova/+/790447 is the wallabyt backport taht will adress this for ml2/ovs

Reviewed: https://review.opendev.org/c/openstack/nova/+/797142
Committed: https://opendev.org/openstack/nova/commit/99cf5292c782828ea33a3448ae0e7da028d5d176
Submitter: "Zuul (22348)"
Branch: master

commit 99cf5292c782828ea33a3448ae0e7da028d5d176
Author: Stephen Finucane <email address hidden>
Date: Fri Jun 18 17:55:50 2021 +0100

    objects: Fix VIFMigrateData.supports_os_vif_delegation setter

    We're using a 'profile' getter/setter in the 'VIFMigrateData' object to
    ensure we transform the JSON encoded string we store in the database to
    an actual dictionary on load and vice versa on save. However, because
    the getter is returning a new object (constructed from 'json.loads')
    rather than a reference to the original data (which is a string),
    modifying this object doesn't actually change the underlying data in the
    object. We were relying on this broken behavior to set the
    'supports_os_vif_delegation' attribute of the 'VIFMigrateData' object
    and trigger the delegated port creation introduced in change
    I11fb5d3ada7f27b39c183157ea73c8b72b4e672e, which means that code isn't
    actually doing anything yet. Resolve this.

    Change-Id: I362deb1088c88cdcd8219922da9dc9a01b10a940
    Signed-off-by: Stephen Finucane <email address hidden>
    Related-Bug: #1734320
    Related-Bug: #1815989

Reviewed: https://review.opendev.org/c/openstack/nova/+/797428
Committed: https://opendev.org/openstack/nova/commit/fa0fb2fe3d61de1cb871c48ee97053cf2fb5827a
Submitter: "Zuul (22348)"
Branch: master

commit fa0fb2fe3d61de1cb871c48ee97053cf2fb5827a
Author: Stephen Finucane <email address hidden>
Date: Tue Jun 22 11:37:22 2021 +0100

    libvirt: Always delegate OVS plug to os-vif

    In change I11fb5d3ada7f27b39c183157ea73c8b72b4e672e, we started
    delegating plugging of OVS ports to os-vif to work around a number of
    bugs. However, this was only introduced for live migration. Plugging is
    still handled by libvirt for spawn. This results in an odd situation,
    whereby an interface of type 'bridge' will be use when creating the
    instance initially, only for this to change to 'ethernet' on live
    migration. Resolve this by *always* delegating plugging to os-vif. This
    is achieved by consistently setting the 'delegate_create' attribute of
    'nova.network.model.VIF' to 'True', which will later get transformed to
    the 'create_port' attribute of the 'os_vif.objects.vif.VIFOpenVSwitch'
    object(s) created in 'nova.network.os_vif_util._nova_to_osvif_vif_ovs'
    and ultimately result in delegate port creation.

    Note that we don't need to worry about making the setting of
    'delegate_create' conditional on whether we're looking at an OVS port or
    not: this will be handled by '_nova_to_osvif_vif_ovs'. We also don't
    need to worry about unsetting this attribute before a live migration:
    the 'delegate_create' attribute is always overridden as part of
    'nova.objects.migrate_data.VIFMigrateData.get_dest_vif'.

    Change-Id: I014c5a81752f86c6b99d19d769c42f318e18e676
    Signed-off-by: Stephen Finucane <email address hidden>
    Related-Bug: #1734320
    Related-Bug: #1815989

Reviewed: https://review.opendev.org/c/openstack/nova/+/770844
Committed: https://opendev.org/openstack/nova/commit/c0a36d917794fed77e75ba9ed853c01a77b540bd
Submitter: "Zuul (22348)"
Branch: stable/train

commit c0a36d917794fed77e75ba9ed853c01a77b540bd
Author: Sean Mooney <email address hidden>
Date: Wed Dec 16 13:12:13 2020 +0000

    only wait for plugtime events in pre-live-migration

    This change modifies _get_neutron_events_for_live_migration
    to filter the event to just the subset that will be sent
    at plug-time.

    Currently neuton has a bug where by the dhcp agent
    send a network-vif-plugged event during live migration after
    we update the port profile with "migrating-to:"
    this cause a network-vif-plugged event to be sent for
    configuration where vif_plugging in nova/os-vif is a noop.

    when that is corrected the current logic in nova cause the migration
    to time out as its waiting for an event that will never arrive.

    This change filters the set of events we wait for to just the plug
    time events.

    Conflicts:
        nova/compute/manager.py
        nova/tests/unit/compute/test_compute_mgr.py

    Related-Bug: #1815989
    Closes-Bug: #1901707
    Change-Id: Id2d8d72d30075200d2b07b847c4e5568599b0d3b
    (cherry picked from commit 8b33ac064456482158b23c2a2d52f819ebb4c60e)
    (cherry picked from commit ef348c4eb3379189f290217c9351157b1ebf0adb)
    (cherry picked from commit d9c833d5a404dfa206e08c97543e80cb613b3f0b)

Hello All,
In the meantime that an official solution will be ready, as regards kolla wallaby, I have created a service that does the discovery of instances on each compute node and, finding a new one, announces it with the command:
docker exec nova_libvirt virsh qemu-monitor-command $ instance --hmp announce_self.

Using the above procedure I lose 2/3 packets during live migration.
Ignazio

this is already fixed on master we need to do the backport but at this point that is all that is left. its also fix in neutron at this point. the neutron backport has alreay been done i belive. to train. i have also fixed ovn for kernel ovs and smubmitted patches for vhost-user with ovn last weeks.

docker exec nova_libvirt virsh qemu-monitor-command $ instance --hmp announce_self. will be a semi effective workaround when using ml2/ovs. it will have no effect for ovn as it has a different unerlying cause specificaly it looses packets because the ovs flows are not installed to sending arp packets does not help with that.

This issue was fixed in the openstack/nova 24.0.0.0rc1 release candidate.

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/c/openstack/nova/+/820682

Reviewed: https://review.opendev.org/c/openstack/nova/+/741529
Committed: https://opendev.org/openstack/nova/commit/d44e24efe28e825fbfd2c75a032bf2d10109a439
Submitter: "Zuul (22348)"
Branch: master

commit d44e24efe28e825fbfd2c75a032bf2d10109a439
Author: Tobias Urdin <email address hidden>
Date: Thu Jul 16 21:29:32 2020 +0200

    libvirt: Add announce-self post live-migration workaround

    This patch adds a workaround that can be enabled
    to send an announce_self QEMU monitor command
    post live-migration to send out RARP frames
    that was lost due to port binding or flows not
    being installed.

    Please note that this makes marks the domain
    in libvirt as tainted.

    See previous information about this issue in
    the [1] bug.

    [1] https://bugs.launchpad.net/nova/+bug/1815989

    Change-Id: I7a6a6fe5f5b23e76948b59a85ca9be075a1c2d6d
    Related-Bug: 1815989

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/825064

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/825175

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/825176

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/825177

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/825178

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/828148

Reviewed: https://review.opendev.org/c/openstack/nova/+/825064
Committed: https://opendev.org/openstack/nova/commit/a8981422afdd09f8cfea053e592c15e771fbe969
Submitter: "Zuul (22348)"
Branch: stable/xena

commit a8981422afdd09f8cfea053e592c15e771fbe969
Author: Tobias Urdin <email address hidden>
Date: Thu Jul 16 21:29:32 2020 +0200

    libvirt: Add announce-self post live-migration workaround

    NOTE(melwitt): This is the combination of two commits, the workaround
    config option and a followup change to add a note that enabling the
    workaround will cause the guest domain to be considered tainted by
    libvirt.

    This patch adds a workaround that can be enabled
    to send an announce_self QEMU monitor command
    post live-migration to send out RARP frames
    that was lost due to port binding or flows not
    being installed.

    Please note that this makes marks the domain
    in libvirt as tainted.

    See previous information about this issue in
    the [1] bug.

    [1] https://bugs.launchpad.net/nova/+bug/1815989

    Related-Bug: 1815989

    Update announce self workaround opt description

    This updates the announce self workaround config opt
    description to include info about instance being set
    as tainted by libvirt.

    Change-Id: I8140c8fe592dd54fc09a9510723892806db49a56
    (cherry picked from commit 2aa1ed5810b67b9a8f18b2ec5e21004f93831168)

    Change-Id: I7a6a6fe5f5b23e76948b59a85ca9be075a1c2d6d
    (cherry picked from commit d44e24efe28e825fbfd2c75a032bf2d10109a439)

Reviewed: https://review.opendev.org/c/openstack/nova/+/825178
Committed: https://opendev.org/openstack/nova/commit/9609ae0bab30675e184d1fc63aec849c1de020d0
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 9609ae0bab30675e184d1fc63aec849c1de020d0
Author: Tobias Urdin <email address hidden>
Date: Thu Jul 16 21:29:32 2020 +0200

    libvirt: Add announce-self post live-migration workaround

    NOTE(melwitt): This is the combination of two commits, the workaround
    config option and a followup change to add a note that enabling the
    workaround will cause the guest domain to be considered tainted by
    libvirt.

    This patch adds a workaround that can be enabled
    to send an announce_self QEMU monitor command
    post live-migration to send out RARP frames
    that was lost due to port binding or flows not
    being installed.

    Please note that this makes marks the domain
    in libvirt as tainted.

    See previous information about this issue in
    the [1] bug.

    [1] https://bugs.launchpad.net/nova/+bug/1815989

    Related-Bug: 1815989

    Update announce self workaround opt description

    This updates the announce self workaround config opt
    description to include info about instance being set
    as tainted by libvirt.

    Change-Id: I8140c8fe592dd54fc09a9510723892806db49a56
    (cherry picked from commit 2aa1ed5810b67b9a8f18b2ec5e21004f93831168)

    Change-Id: I7a6a6fe5f5b23e76948b59a85ca9be075a1c2d6d
    (cherry picked from commit d44e24efe28e825fbfd2c75a032bf2d10109a439)
    (cherry picked from commit a8981422afdd09f8cfea053e592c15e771fbe969)

Change abandoned by "Stephen Finucane <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/797316

Change abandoned by "Stephen Finucane <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/797291

Change abandoned by "Stephen Finucane <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/797144

Change abandoned by "Stephen Finucane <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/790447

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/stein
Review: https://review.opendev.org/c/openstack/nova/+/820682
Reason: This branch transitioned to End of Life for this project, open patches needs to be closed to be able to delete the branch.

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/754475
Reason: If needed, please restore the patch and address the comments.

Reviewed: https://review.opendev.org/c/openstack/nova/+/790447
Committed: https://opendev.org/openstack/nova/commit/23a4b27dc0c156ad6cbe5260d518da3fd62294b8
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 23a4b27dc0c156ad6cbe5260d518da3fd62294b8
Author: Stephen Finucane <email address hidden>
Date: Fri Apr 30 12:51:35 2021 +0100

    libvirt: Delegate OVS plug to os-vif

    os-vif 1.15.0 added the ability to create an OVS port during plugging
    by specifying the 'create_port' attribute in the 'port_profile' field.
    By delegating port creation to os-vif, we can rely on it's 'isolate_vif'
    config option [1] that will temporarily configure the VLAN to 4095
    (0xfff), which is reserved for implementation use [2] and is used by
    neutron to as a dead VLAN [3]. By doing this, we ensure VIFs are plugged
    securely, preventing guests from accessing other tenants' networks
    before the neutron OVS agent can wire up the port.

    This change requires a little dance as part of the live migration flow.
    Since we can't be certain the destination host has a version of os-vif
    that supports this feature, we need to use a sentinel to indicate when
    it does. Typically we would do so with a field in
    'LibvirtLiveMigrateData', such as the 'src_supports_numa_live_migration'
    and 'dst_supports_numa_live_migration' fields used to indicate support
    for NUMA-aware live migration. However, doing this prevents us
    backporting this important fix since o.vo changes are not backportable.
    Instead, we (somewhat evilly) rely on the free-form nature of the
    'VIFMigrateData.profile_json' string field, which stores JSON blobs and
    is included in 'LibvirtLiveMigrateData' via the 'vifs' attribute, to
    transport this sentinel. This is a hack but is necessary to work around
    the lack of a free-form "capabilities" style dict that would allow us do
    backportable fixes to live migration features.

    Note that this change has the knock on effect of modifying the XML
    generated for OVS ports: when hybrid plug is false will now be of type
    'ethernet' rather than 'bridge' as before. This explains the larger than
    expected test damage but should not affect users.

    Changes:
      lower-constraints.txt
      requirements.txt
      nova/network/os_vif_util.py
      nova/tests/unit/virt/libvirt/test_vif.py
      nova/tests/unit/virt/libvirt/test_driver.py
      nova/virt/libvirt/driver.py

    NOTE(stephenfin): Change I362deb1088c88cdcd8219922da9dc9a01b10a940
    ("objects: Fix VIFMigrateData.supports_os_vif_delegation setter") which
    contains an important fix for the original change, is squashed into this
    change. In addition, the os-vif version bump we introduced in the
    original version of this patch is not backportable and as a result, we
    must introduce two additional checks. Both checks ensure we have a
    suitable version of os-vif and skip the new code paths if not. The first
    check is in the libvirt driver's 'check_can_live_migrate_destination'
    function, which as the name suggests runs on the destination host early
    in the live migration process. If os-v...

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/825177
Reason: stable/train branch of nova projects' have been tagged as End of Life. All open patches have to be abandoned in order to be able to delete the branch.