OpenStack Compute (nova)

device_path not saved into bdm.connection_info during pre_live_migration

  1. Series victoria
  2. Bug #1939545
Bug #1939545 reported by Lee Yarwood
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Lee Yarwood
Train
In Progress
Undecided
Unassigned
Ussuri
In Progress
Undecided
Unassigned
Victoria
In Progress
Undecided
Unassigned
Wallaby
Fix Released
Undecided
Unassigned

Bug Description

Description
===========

Various block based volume drivers attempt to save a device_path back into the stashed connection_info stored within Nova's block device mappings *after* connecting a volume to the underlying host. Thanks to the indirection caused by the various data structures used between the virt and compute layers this isn't actually saved into the underlying block device mapping database record during a typical attach flow until we get back into the compute and driver block device layer:

https://github.com/openstack/nova/blob/84b61790763f91e12eebb96d955e2f83abc00d56/nova/virt/block_device.py#L613-L619

However when an instance is being live migrated these volumes are connected as part of pre_live_migration on the destination and no attempt is made to save the updates made to the connection_info of the volume into the database. This isn't a massive problem as os-brick can for the most part lookup the device during future operations but it is obviously inefficient.

This was initially hit in bug #1936439 but that bug is now being used to track in a trivial DEBUG log change while this bug will track in the underlying fix for the above issue.

Steps to reproduce
==================
* Attach an iSCSI/FC/NVME etc volume to an instance
* Live migrate the instance
* Confirm that device_path isn't present in the connection_info stashed in the bdm

Expected result
===============
device_path is stashed in the connection_info of the bdm

Actual result
=============
device_path isn't stashed in the connection_info of the bdm

Environment
===========
1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/

master

2. Which hypervisor did you use?
   (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
   What's the version of that?

libvirt

2. Which storage type did you use?
   (For example: Ceph, LVM, GPFS, ...)
   What's the version of that?

LVM/iSCSI/FC/NVMe, any block based volume backends.

3. Which networking type did you use?
   (For example: nova-network, Neutron with OpenVSwitch, ...)

N/A

Logs & Configs
==============

See bug #1936439

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/804229

Changed in nova:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/804230

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/804229
Committed: https://opendev.org/openstack/nova/commit/268b7169554f871d3813d78e79861934d086484d
Submitter: "Zuul (22348)"
Branch: master

commit 268b7169554f871d3813d78e79861934d086484d
Author: Lee Yarwood <email address hidden>
Date: Wed Aug 11 10:51:33 2021 +0100

    Add a regression test for bug 1939545

    Assert the behaviour of the libvirt driver during pre_live_migration
    with instances that have block based volumes attached.

    Bug #1939545 covering the case where the returned path from os-brick
    isn't being saved into the connection_info of the associated bdm in
    Nova.

    The cinder fixture is extended as part of this change to offer
    connection_info listing the iscsi volume type in order to correctly
    assert the behaviour of the associated libvirt volume driver.

    Related-Bug: #1939545
    Change-Id: I3f1073d70e4332737e33f3c9c00cf129d0ef76b4

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/804230
Committed: https://opendev.org/openstack/nova/commit/962eda94d52321c3237da870c3d0455c6f0e851b
Submitter: "Zuul (22348)"
Branch: master

commit 962eda94d52321c3237da870c3d0455c6f0e851b
Author: Lee Yarwood <email address hidden>
Date: Wed Aug 11 12:04:11 2021 +0100

    compute: Ensure updates to bdms during pre_live_migration are saved

    When connecting volumes to the underlying host the virt drivers can
    attempt to stash additional metadata returned from os-brick into the
    connection_info associated with a bdm. This pretty janky behaviour
    relies on someone later calling .save() against the underlying
    BlockDeviceMapping object to persist these changes into the database as
    happens in the driver block device layer during a standard volume
    attach.

    However during pre_live_migration no call was made to .save() resulting
    in the changes made to the connection_info being lost. This change
    simply introduces this call at the end of the pre_live_migration method
    on the destination via the driver bdm objects we provide in
    block_device_info.

    Closes-Bug: #1939545
    Change-Id: Iea8896682fc28e3a5cd25afa45238272bee745e1

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 24.0.0.0rc1

This issue was fixed in the openstack/nova 24.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/843679

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/843680

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/843702

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/victoria)

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/843948

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/843949

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/ussuri)

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/843951

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/843952

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/843959

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/843960

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/wallaby)

Change abandoned by "Artom Lifshitz <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/843702

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/nova/+/843679
Committed: https://opendev.org/openstack/nova/commit/9f7b81c4cfe9f6a41fb28ec3b9d20616817ceab5
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 9f7b81c4cfe9f6a41fb28ec3b9d20616817ceab5
Author: Lee Yarwood <email address hidden>
Date: Wed Aug 11 10:51:33 2021 +0100

    Add a regression test for bug 1939545

    Assert the behaviour of the libvirt driver during pre_live_migration
    with instances that have block based volumes attached.

    Bug #1939545 covering the case where the returned path from os-brick
    isn't being saved into the connection_info of the associated bdm in
    Nova.

    The cinder fixture is extended as part of this change to offer
    connection_info listing the iscsi volume type in order to correctly
    assert the behaviour of the associated libvirt volume driver.

    Conflicts: nova/tests/fixtures/cinder.py did not exist, applied
    changes to CinderFixture class in nova/tests/fixtures.py. Also:
    * Switched to using uuidsentinel instead of uuids, as the import is
      done as the former in stable/wallaby.
    * Fixed the fakelibvirt import to match where it lives in
      stable/wallaby.
    * Replace use of OSBrickFixture() that didn't exist in stable/wallaby
      with the fake_os_brick_connector from the libvirt unit tests.
    * Fix fakelibvirt migrateToURI3 monkeypatch to use correct path for
      stable/wallaby.

    Related-Bug: #1939545
    Change-Id: I3f1073d70e4332737e33f3c9c00cf129d0ef76b4
    (cherry picked from commit 268b7169554f871d3813d78e79861934d086484d)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/nova/+/843680
Committed: https://opendev.org/openstack/nova/commit/9bccc5a04354b0490fbe7e0f0b98cf55d4b7c9c4
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 9bccc5a04354b0490fbe7e0f0b98cf55d4b7c9c4
Author: Lee Yarwood <email address hidden>
Date: Wed Aug 11 12:04:11 2021 +0100

    compute: Ensure updates to bdms during pre_live_migration are saved

    When connecting volumes to the underlying host the virt drivers can
    attempt to stash additional metadata returned from os-brick into the
    connection_info associated with a bdm. This pretty janky behaviour
    relies on someone later calling .save() against the underlying
    BlockDeviceMapping object to persist these changes into the database as
    happens in the driver block device layer during a standard volume
    attach.

    However during pre_live_migration no call was made to .save() resulting
    in the changes made to the connection_info being lost. This change
    simply introduces this call at the end of the pre_live_migration method
    on the destination via the driver bdm objects we provide in
    block_device_info.

    Closes-Bug: #1939545
    Change-Id: Iea8896682fc28e3a5cd25afa45238272bee745e1
    (cherry picked from commit 962eda94d52321c3237da870c3d0455c6f0e851b)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 23.2.2

This issue was fixed in the openstack/nova 23.2.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/train)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/843960
Reason: stable/train branch of nova projects' have been tagged as End of Life. All open patches have to be abandoned in order to be able to delete the branch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/843959
Reason: stable/train branch of nova projects' have been tagged as End of Life. All open patches have to be abandoned in order to be able to delete the branch.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.