Instances with hardware offloaded ovs ports lose access after failed live migrations

https://docs.openstack.org/neutron/latest/admin/config-sriov.html#known-limitations
This doc shows that before openstack Train release, Live migration with sriov port was not supported. After T release, please following the note to ensure that you have the correct settings of the guest interface.

"""
Indirect mode SR-IOV interfaces (vnic-type: macvtap or virtio-forwarder) can now be migrated transparently to the guest. Direct mode SR-IOV interfaces (vnic-type: direct or direct-physical) are detached before the migration and reattached after the migration so this is not transparent to the guest. To avoid loss of network connectivy when live migrating with direct mode sriov the user should create a failover bond in the guest with a transparently live migration port type e.g. vnic-type normal or indirect mode SR-IOV.
"""

Hi folks, thanks for triaging this. The environment we are running is Focal+Ussuri, so should it
support the live migration.

This is the port configuration:

ubuntu@cloud-benchmarking:~$ openstack port show 3a3329b5-d123-4009-a152-3e1ed33bd15f
+-------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| allowed_address_pairs | |
| binding_host_id | node-laveran.maas |
| binding_profile | capabilities='['switchdev'], pci_slot='0000:03:0a.7', pci_vendor_info='15b3:1018', physical_network=''', pci_slot='0000:03:04.1', pci_vendor_info='15b3:1018', physical_network= |
| binding_vif_details | bridge_name='br-int', connectivity='l2', datapath_type='system', ovs_hybrid_plug='True', port_filter='True' |
| binding_vif_type | ovs |
| binding_vnic_type | direct |
| created_at | 2021-09-16T15:32:32Z |
| data_plane_status | None |
| description | |
| device_id | de492a45-6498-4313-be8d-9961c53f78d9 |
| device_owner | compute:nova ...

Hello:

The problem reported relates to Nova migration process. When an error like this one occurs, Nova should rebind again the port in the source host. If this is not happening, as you described, then Nova process should be reviewed.

Please, provide the Neutron logs (SRIOV agent, Neutron server) in DEBUG mode if possible.

Regards.

Hi Rodolfo,

Good point. Indeed the rollback + cleanup process is something that nova should be doing.
From the Neutron side, AFAIC, everything is working as it should.

Erlon

i need to read this a few times to make sure i have not missed anything but i just want to level set on the current support

today the following reqruiements and restriciton apply to the use of sriov and/or ovs hardware offload.

1 if the sriov nic agent is used for standard sriov vnic types (direct, direct-physical, macvtap) nics __must not__ be in __switchdev__ mode, __must__ be in __legacy__ mode
2 vdpa support in nova current does not support any move operations, vdpa supprot in nova requeres teh nic to be in switchdev mode.
3 hardware offloaded ovs uses the ml2/ovs or ml2/ovn mechansiume drivers and does nto use the sriov nic agent.
4 we do not support using the sriov nic agent and ovs hardware offload on teh same phsyical nic
  when using the sriov-nic-agent the nic must be in legacy mode and when using hardware offload it must be in swtichdev mode

we do not currently test or offically support sriov live migration between different ml2 driver but in princial it can work but any work required to make it work would really be a new feature.
it was not in the scope of the sriov live migration spec. limited issue could be adress as bug fixes but live migration form host using sriov nic agent to hardware offlowaded ovs was not in scope.

with that understanding set lets look at the bug you have reported

Check the logs for: libvirt.libvirtError: Requested operation is not valid: PCI device 0000:03:04.1 is in use by driver QEMU, domain instance-00000001

this is cause by trying to do move other vms that have neutron sriov port with shelve and unshleve https://bugs.launchpad.net/nova/+bug/1851545

this is very unlikely to be related to sriov live migration feature.

looking at the first pastbin you provdied you are not using standard sriov you are using hardware offloaded ovs at least on the destination host.

 Successfully unplugged vif VIFHostDevice(active=True,address=fa:16:3e:4d:86:24,dev_address=0000:03:04.1,dev_type='ethernet',has_traffic_filtering=True,id=3a3329b5-d123-4009-a152-3e1ed33bd15f,network=Network(725fffcf-f6bf-4e6f-8430-ca2536311805),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True)

can you confirm that you are using hardware offloade ovs on both the source and destionaton hosts
and if you are using ml2/ovs or ml2/ovn. The info in commnet 2 also indicates this is not
starndard sriov but hardware offloaded ovs which has a very differnt code path in nova and neutron so we should fix the title if this is infact hardware offloaded ovs.

by the way nova does not tear down any networking on the source node until post_live migration and it does not activate the dest port bidnign until post live migration either so nova intentiolly leave the source node fully intack until post live migration so that on revert or errors in pre_live migration no reconfirutation is needed on the source node and i can simple delete the inactive port binding on the dest node.

so in a rollback case nova should not need to do any rebinding of the port on the source node.

what i suspect has happend here is the live migration fails in the migration phase
after pre_live_migrate

in that case in most case setting the vm to error is actully the expected outcome.
we can role back in some case but we do not garunetee that roleback will happen in migrate.

i suspect that the reason connectivity is lost is because the failure to migrate due to a vm on the destination using the incorrect VF happens in driver.live_migration after we have unpluged the pci devices form the source vm which takes use to the _rollback_live_migration codepath but that has two outcomes. it can set teh vm to error or in limited cases rollback and leave the vm running on the source node unaffected by the failed live migration.

https://docs.openstack.org/nova/latest/reference/live-migration.html

in this case i suspect we don reattech the VF to the vm.

this is where we detach the interfaces

https://github.com/openstack/nova/blob/66574018b517f14dc26e581d0ddaa7788806f83e/nova/virt/libvirt/driver.py#L9602-L9625

in _live_migration_operation then we start the migration here
https://github.com/openstack/nova/blob/66574018b517f14dc26e581d0ddaa7788806f83e/nova/virt/libvirt/driver.py#L9662-L9667

in _rollback_live_migration we do correctly revert the pci allcoations
https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L9083-L9093

and we call back into the driver to its revert

https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L9115-L9116

and that reattached the interface to the vms
https://github.com/openstack/nova/blob/66574018b517f14dc26e581d0ddaa7788806f83e/nova/virt/libvirt/driver.py#L10127-L10137

unless you have correctly configured network manager in the guest to retrigger on the hotplug of the interface the guest wont have network connectivty restored until it reboots and the on boot network configurtion scripts run.

we have specifcaly catered for this rollback case in the code
https://github.com/openstack/nova/blob/66574018b517f14dc26e581d0ddaa7788806f83e/nova/virt/libvirt/driver.py#L10110-L10125
so i am not seeing anythign that woudl point to a nova bug currently.

the libvirt error simply looks like you had instance that were affected by https://bugs.launchpad.net/nova/+bug/1851545 which will only be noticed if you try to start new instance on host ver the vms are not using the pci device that is claimed in the database.

Hi Sean,

Thanks a lot for the thorough review and evaluation of the bug.
I appreciated it! It took me a while so I could get time to parse and
get you a proper response.

> 1 if the sriov nic agent is used for standard sriov vnic types (direct,
> direct-physical, macvtap) nics __must not__ be in __switchdev__ mode,
> __must__ be in __legacy__ mode

Not sure exactly what you mean here but, there is only one agent
(openvswitch-agent) on the computes and network nodes. That agent uses
the configuration as in [2] and is not configured as SRIOV, the
switchdev/hw offloading configuration is done in openvswitch.

> 2 vdpa support in nova current does not support any move operations,
> vdpa support in nova requires the nic to be in switchdev mode.

I don't believe we are using this.

> 3 hardware offloaded ovs uses the ml2/ovs or ml2/ovn mechansiume
> drivers and does not use the sriov nic agent.

Right, this is how we are doing.

> 4 we do not support using the sriov nic agent and ovs hardware offload
> on the same phsyical nic when using the sriov-nic-agent the nic must be
> in legacy mode and when using hardware offload it must be in swtichdev
> mode live migration form host using sriov nic agent to hardware
> offlowaded ovs was not in scope.

The migration is beteween 2 switchdev hosts with ml2/ovs

> this is cause by trying to do move other vms that have neutron sriov
> port with shelve and unshleve
> https://bugs.launchpad.net/nova/+bug/1851545

The bug above might be one of the possible problems related to this
message. If you follow the logs[3] you will see that here, this is
happening because:

1 - During pre_live_migration, the neutron port is attached on the
    destination host[4]
2 - pre_live_migration fails on the destination host and triggers an
    exception on the source host[5] [6]->[7]
3 - rollback is triggered and tries re-attach the port to source host,
    but QEMU instance still holds the PCI address[8] and the PCI message
    error is triggered

> what i suspect has happend here is the live migration fails in the
  migration phase after pre_live_migrate

As I mentioned above, the failure was in the pre_live_migration funcion
(I caused in in my env, but it happeneded for some reason at the customer
site)

> unless you have correctly configured network manager in the guest to
> retrigger on the hotplug of the interface the guest wont have network
> connectivty restored until it reboots and the on boot network
> configurtion scripts run.

So, these are standard ubuntu images and are for sure configured to
hotplug given they don't loose connection when the migration works.

So, it seems that the bug here is that rollback_live_migration_at_source()
is called for both when the migration fails on pre_live_migration()
or live_migration. But, for the case when something fails on
pre_live_migration, this shouldn't be done.

Now I'm curious and will test if this attempt/error to re-attach the
device in the same address is the thing that is making the instance to
loose conectivity. I'll test that. Please let me know your thoguths on
my suspition above.

Erlon
_____________________
[1] neutron.conf: https://gist.github.com/sombrafam/ca6...

I have tested to remove the interface detaching from the code and the instance doesn't loose
connectivity after a failed migration. I have a draft here:

https://review.opendev.org/c/openstack/nova/+/815324

Reviewed: https://review.opendev.org/c/openstack/nova/+/821840
Committed: https://opendev.org/openstack/nova/commit/d43538712c034023bdb3e25cd7adfdee409ed596
Submitter: "Zuul (22348)"
Branch: master

commit d43538712c034023bdb3e25cd7adfdee409ed596
Author: Erlon R. Cruz <email address hidden>
Date: Tue Dec 7 17:51:53 2021 -0300

    Adds regression test for bug LP#1944619

    Related-bug: #1944619
    Change-Id: Ia208ef0dd0bab1c59d025234f4da7537c39cda61

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/833079

Change abandoned by "Balazs Gibizer <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/nova/+/833079
Reason: We went to reverting the original patch instead with https://review.opendev.org/c/openstack/nova/+/832902

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/833166

Since we reverted the patch and as the bug is not a regression, I'm removing the yoga-rc-potential flag.

Hey Sylvian,

Can you revisit the patches[1] and re-consider the addition to yoga? I've refactored the tests and run exhaustive testing on them to make sure they are stable.

[1] https://review.opendev.org/q/topic:bug%252F1944619

Test result comparations here:

[good] https://gist.github.com/sombrafam/03112dea45b3c6823eae94e47bc7511e
[bad] https://gist.github.com/sombrafam/3088a3f014dc00e69925b96c351feaf5

Erlon

Reviewed: https://review.opendev.org/c/openstack/nova/+/833166
Committed: https://opendev.org/openstack/nova/commit/2ddb8bf53fdf9a17c09afc4987ab6efe8ba97696
Submitter: "Zuul (22348)"
Branch: master

commit 2ddb8bf53fdf9a17c09afc4987ab6efe8ba97696
Author: Erlon R. Cruz <email address hidden>
Date: Thu Mar 10 15:50:54 2022 -0300

    Adds regression test for bug LP#1944619

    Related-bug: #1944619
    Closes-bug: #1964472
    Change-Id: Ie7e5377aea23a4fbd7ad91f245d17def6d0fb927

Reviewed: https://review.opendev.org/c/openstack/nova/+/815324
Committed: https://opendev.org/openstack/nova/commit/63ffba7496182f6f6f49a380f3c639fc3ded9772
Submitter: "Zuul (22348)"
Branch: master

commit 63ffba7496182f6f6f49a380f3c639fc3ded9772
Author: Erlon R. Cruz <email address hidden>
Date: Tue Dec 7 17:39:58 2021 -0300

    Fix pre_live_migration rollback

    During the pre live migration process, Nova performs most of the
    tasks related to the creation and operation of the VM in the destination
    host. That is done without interrupting any of the hardware in the source
    host. If the pre_live_migration fails, those same operations should be
    rolled back.

    Currently nova is sharing the _rollback_live_migration for both
    live and pre_live migration rollbacks, and that is causing the source
    host to try to re-attach network interfaces on the source host where
    they weren't actually de-attached.

    This patch fixes that by adding a conditional to allow nova to do
    different paths for migration and pre_live_migration rollbacks.

    Closes-bug: #1944619
    Change-Id: I784190ac356695dd508e0ad8ec31d8eaa3ebee56

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/nova/+/836014

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/836015

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/836016

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/836017

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/836018

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/838323

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/838332

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/838334

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/838550

Change abandoned by "Erlon R. Cruz <email address hidden>" on branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/838323
Reason: Wrong code

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/nova/+/838788

Reviewed: https://review.opendev.org/c/openstack/nova/+/838788
Committed: https://opendev.org/openstack/nova/commit/3402aa7a53a48e6abc0fb8a2620cd449fd3f75fe
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 3402aa7a53a48e6abc0fb8a2620cd449fd3f75fe
Author: Erlon R. Cruz <email address hidden>
Date: Thu Mar 10 15:50:54 2022 -0300

    Adds regression test for bug LP#1944619

    Related-bug: #1944619
    Closes-bug: #1964472
    Change-Id: Ie7e5377aea23a4fbd7ad91f245d17def6d0fb927
    (cherry picked from commit 2ddb8bf53fdf9a17c09afc4987ab6efe8ba97696)

Reviewed: https://review.opendev.org/c/openstack/nova/+/836014
Committed: https://opendev.org/openstack/nova/commit/29b94aa34ad954e617c2a0d6df0809765dced188
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 29b94aa34ad954e617c2a0d6df0809765dced188
Author: Erlon R. Cruz <email address hidden>
Date: Tue Dec 7 17:39:58 2021 -0300

    Fix pre_live_migration rollback

    During the pre live migration process, Nova performs most of the
    tasks related to the creation and operation of the VM in the destination
    host. That is done without interrupting any of the hardware in the source
    host. If the pre_live_migration fails, those same operations should be
    rolled back.

    Currently nova is sharing the _rollback_live_migration for both
    live and pre_live migration rollbacks, and that is causing the source
    host to try to re-attach network interfaces on the source host where
    they weren't actually de-attached.

    This patch fixes that by adding a conditional to allow nova to do
    different paths for migration and pre_live_migration rollbacks.

    Closes-bug: #1944619
    Change-Id: I784190ac356695dd508e0ad8ec31d8eaa3ebee56
    (cherry picked from commit 63ffba7496182f6f6f49a380f3c639fc3ded9772)

Reviewed: https://review.opendev.org/c/openstack/nova/+/838550
Committed: https://opendev.org/openstack/nova/commit/1059921383060257b3a66596dec96a04b208dea9
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 1059921383060257b3a66596dec96a04b208dea9
Author: Erlon R. Cruz <email address hidden>
Date: Thu Mar 10 15:50:54 2022 -0300

    Adds regression test for bug LP#1944619

    Related-bug: #1944619
    Closes-bug: #1964472
    Change-Id: Ie7e5377aea23a4fbd7ad91f245d17def6d0fb927
    (cherry picked from commit 2ddb8bf53fdf9a17c09afc4987ab6efe8ba97696)
    (cherry picked from commit 3402aa7a53a48e6abc0fb8a2620cd449fd3f75fe)

Reviewed: https://review.opendev.org/c/openstack/nova/+/836015
Committed: https://opendev.org/openstack/nova/commit/90c51902e9ff9fcc23cba00904a0a5d9db6a5d6a
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 90c51902e9ff9fcc23cba00904a0a5d9db6a5d6a
Author: Erlon R. Cruz <email address hidden>
Date: Tue Dec 7 17:39:58 2021 -0300

    Fix pre_live_migration rollback

    During the pre live migration process, Nova performs most of the
    tasks related to the creation and operation of the VM in the destination
    host. That is done without interrupting any of the hardware in the source
    host. If the pre_live_migration fails, those same operations should be
    rolled back.

    Currently nova is sharing the _rollback_live_migration for both
    live and pre_live migration rollbacks, and that is causing the source
    host to try to re-attach network interfaces on the source host where
    they weren't actually de-attached.

    This patch fixes that by adding a conditional to allow nova to do
    different paths for migration and pre_live_migration rollbacks.

    Closes-bug: #1944619
    Change-Id: I784190ac356695dd508e0ad8ec31d8eaa3ebee56
    (cherry picked from commit 63ffba7496182f6f6f49a380f3c639fc3ded9772)
    (cherry picked from commit 29b94aa34ad954e617c2a0d6df0809765dced188)

This issue was fixed in the openstack/nova 25.0.1 release.

This issue was fixed in the openstack/nova 24.1.1 release.

This issue was fixed in the openstack/nova 26.0.0.0rc1 release candidate.

Change abandoned by "Erlon R. Cruz <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/836018
Reason: Not backporting this anymore as is not that critical and is causing merge conflicts.

Change abandoned by "Erlon R. Cruz <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/838334
Reason: Not backporting this anymore as is not that critical and is causing merge conflicts.

Change abandoned by "Erlon R. Cruz <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/836017
Reason: Not backporting this anymore as is not that critical and is causing merge conflicts.

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/862401

Change abandoned by "Erlon R. Cruz <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/862401
Reason: Abandoning in favor of Change Ia208ef0dd0bab1c59d025234f4da7537c39cda61

Change abandoned by "Erlon R. Cruz <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/836016
Reason: Not backporting this anymore as is not that critical and is causing failures on the unit tests.

Change abandoned by "Erlon R. Cruz <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/838332
Reason: Not backporting this anymore as is not that critical and is causing failures on the unit tests.