OpenStack Compute (nova)

Duplicate BlockDeviceMapping if attaching volumes too fast

  1. Series rocky
  2. Bug #1937375
Bug #1937375 reported by Felix Huettner
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Felix Huettner
Queens
New
Undecided
Unassigned
Rocky
New
Undecided
Unassigned
Stein
New
Undecided
Unassigned
Train
New
Undecided
Unassigned
Ussuri
New
Undecided
Unassigned
Victoria
New
Undecided
Unassigned
Wallaby
Fix Released
Undecided
Unassigned

Bug Description

Description
===========
When attaching a volume to a running instance the nova-api validates that
the volume is not already attached to the instance. However nova-compute
is responsible for actually creating the BDM entry in the database.
If sending attach requests fast enough it can be possible that the same
"attach_volume" request can be sent to nova-compute for the same
volume/instance combination.

Steps to reproduce
==================
* Create an Instance and a volume
* run in 2 terminals in parallel "openstack server add volume <instance> <volume>"
* when being fast enough this results in "openstack server show" to report the volume id twice

Expected result
===============
The volume is attached to the instance just once

Actual result
=============
The volume is attached to the instance two (or with more parallelity) or more times.

Environment
===========
1. Happens on master till queens (possibly also earlier releases)

2. Which hypervisor did you use?
   Indepentent of Hypervisor, observed with Libvirt

2. Which storage type did you use?
   Independent of Storage, observed with ceph + nfs

3. Which networking type did you use?
   Independent of Network, observed with ovs

Felix Huettner (felix.huettner)
Changed in nova:
assignee: nobody → Felix Huettner (felix.huettner)
OpenStack Infra (hudson-openstack)
Changed in nova:
status: New → In Progress
Revision history for this message
Felix Huettner (felix.huettner) wrote :

Fix is being developed in https://review.opendev.org/c/openstack/nova/+/801990

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/802011
Committed: https://opendev.org/openstack/nova/commit/2ffd9738602531e93495a1feca76bbb687c3e72c
Submitter: "Zuul (22348)"
Branch: master

commit 2ffd9738602531e93495a1feca76bbb687c3e72c
Author: Lee Yarwood <email address hidden>
Date: Fri Jul 23 11:12:56 2021 +0100

    Add functional test for bug 1937375

    Related-Bug: #1937375
    Change-Id: I332d4f33ea6b9506cc24ac12e5c0994f208a3107

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/801990
Committed: https://opendev.org/openstack/nova/commit/2209b0007fe85d7c5439e0bfdfe2120c63898fa2
Submitter: "Zuul (22348)"
Branch: master

commit 2209b0007fe85d7c5439e0bfdfe2120c63898fa2
Author: Felix Huettner <email address hidden>
Date: Fri Jul 23 10:43:32 2021 +0200

    compute: Avoid duplicate BDMs during reserve_block_device_name

    When attaching a volume to a running instance the nova-api validates
    that the volume is not already attached to the instance. However
    nova-compute is responsible for actually creating the BDM entry in the
    database. If sending attach requests fast enough it can be possible
    that the same "attach_volume" request can be sent to nova-compute for
    the same volume/instance combination.

    To work around this we add a check in nova-compute to validate that
    the volume has not been attached in the mean time.

    Closes-Bug: #1937375
    Change-Id: I92f35514efddcb071c7094370b79d91d34c5bc72

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/803717

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/803718

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/victoria)

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/803720

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/803721

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/ussuri)

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/803722

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/803723

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/803727

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/803728

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/stein)

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/c/openstack/nova/+/803762

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/c/openstack/nova/+/803763

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/rocky)

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/c/openstack/nova/+/803764

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/c/openstack/nova/+/803765

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/c/openstack/nova/+/803766

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/c/openstack/nova/+/803767

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 24.0.0.0rc1

This issue was fixed in the openstack/nova 24.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/nova/+/803717
Committed: https://opendev.org/openstack/nova/commit/7a9e3dcd172f187cf93f406b09f49d2ded9bd90d
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 7a9e3dcd172f187cf93f406b09f49d2ded9bd90d
Author: Lee Yarwood <email address hidden>
Date: Fri Jul 23 11:12:56 2021 +0100

    Add functional test for bug 1937375

    Related-Bug: #1937375
    Change-Id: I332d4f33ea6b9506cc24ac12e5c0994f208a3107
    (cherry picked from commit 2ffd9738602531e93495a1feca76bbb687c3e72c)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/nova/+/803718
Committed: https://opendev.org/openstack/nova/commit/2bee83b8a980b2bd9e276b75aa74253f8c0d0a70
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 2bee83b8a980b2bd9e276b75aa74253f8c0d0a70
Author: Felix Huettner <email address hidden>
Date: Fri Jul 23 10:43:32 2021 +0200

    compute: Avoid duplicate BDMs during reserve_block_device_name

    When attaching a volume to a running instance the nova-api validates
    that the volume is not already attached to the instance. However
    nova-compute is responsible for actually creating the BDM entry in the
    database. If sending attach requests fast enough it can be possible
    that the same "attach_volume" request can be sent to nova-compute for
    the same volume/instance combination.

    To work around this we add a check in nova-compute to validate that
    the volume has not been attached in the mean time.

    Closes-Bug: #1937375
    Change-Id: I92f35514efddcb071c7094370b79d91d34c5bc72
    (cherry picked from commit 2209b0007fe85d7c5439e0bfdfe2120c63898fa2)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 23.2.0

This issue was fixed in the openstack/nova 23.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/queens)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/queens
Review: https://review.opendev.org/c/openstack/nova/+/803767
Reason: This branch transitioned to End of Life for this project, open patches needs to be closed to be able to delete the branch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/queens
Review: https://review.opendev.org/c/openstack/nova/+/803766
Reason: This branch transitioned to End of Life for this project, open patches needs to be closed to be able to delete the branch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/rocky)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/rocky
Review: https://review.opendev.org/c/openstack/nova/+/803765
Reason: This branch transitioned to End of Life for this project, open patches needs to be closed to be able to delete the branch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/rocky
Review: https://review.opendev.org/c/openstack/nova/+/803764
Reason: This branch transitioned to End of Life for this project, open patches needs to be closed to be able to delete the branch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/stein)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/stein
Review: https://review.opendev.org/c/openstack/nova/+/803762
Reason: This branch transitioned to End of Life for this project, open patches needs to be closed to be able to delete the branch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/stein
Review: https://review.opendev.org/c/openstack/nova/+/803763
Reason: This branch transitioned to End of Life for this project, open patches needs to be closed to be able to delete the branch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/train)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/803728
Reason: stable/train branch of nova projects' have been tagged as End of Life. All open patches have to be abandoned in order to be able to delete the branch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/803727
Reason: stable/train branch of nova projects' have been tagged as End of Life. All open patches have to be abandoned in order to be able to delete the branch.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.