OpenStack Compute (nova)

detach_volume races with delete

  1. Series rocky
  2. Bug #1662483
Bug #1662483 reported by Matthew Booth
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Gary Kotton
Ocata
Confirmed
Undecided
Unassigned
Pike
Fix Committed
Undecided
Chris Dent
Queens
Fix Committed
Undecided
Chris Dent
Rocky
Fix Committed
Undecided
Chris Dent

Bug Description

If a client does:

nova volume-detach foo vol
nova delete foo

Assuming the volume-detach takes a moment, which it normally does, the delete will race with it also also attempt to detach the same volume. It's possible there are no side effects from this other than untidy log messages, but this is difficult to prove.

I found this looking through CI logs.

Note that volume-detach can also race with other instance operations, including itself. I'm almost certain that if you poke hard enough you'll find some combination that breaks things badly.

Tags: volumes
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/430213

Changed in nova:
assignee: nobody → Matthew Booth (mbooth-9)
status: New → In Progress
Matt Riedemann (mriedem)
tags: added: volumes
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/430213
Reason: This review is > 4 weeks without comment, and is not mergable in it's current state. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message
Sean Dague (sdague) wrote :

There are no currently open reviews on this bug, changing the status back to the previous state and unassigning. If there are active reviews related to this bug, please include links in comments.

Changed in nova:
status: In Progress → New
assignee: Matthew Booth (mbooth-9) → nobody
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/430213
Reason: Nevermind I just updated https://review.openstack.org/#/c/636701/ to add mdbooth as co-author.

Changed in nova:
assignee: nobody → Matt Riedemann (mriedem)
status: New → In Progress
Matt Riedemann (mriedem)
Changed in nova:
assignee: Matt Riedemann (mriedem) → Gary Kotton (garyk)
importance: Undecided → Medium
OpenStack Infra (hudson-openstack)
Changed in nova:
assignee: Gary Kotton (garyk) → Matt Riedemann (mriedem)
Matt Riedemann (mriedem)
Changed in nova:
assignee: Matt Riedemann (mriedem) → Gary Kotton (garyk)
OpenStack Infra (hudson-openstack)
Changed in nova:
assignee: Gary Kotton (garyk) → Matt Riedemann (mriedem)
Matt Riedemann (mriedem)
Changed in nova:
assignee: Matt Riedemann (mriedem) → Gary Kotton (garyk)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/636701
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2e4e203c1a4c4f91dd1aabae8c8d7b745df99be7
Submitter: Zuul
Branch: master

commit 2e4e203c1a4c4f91dd1aabae8c8d7b745df99be7
Author: Gary Kotton <email address hidden>
Date: Wed Feb 13 09:50:14 2019 -0800

    Lock detach_volume

    A detach_volume operation may take place during a attach_volume or
    delete_instance opertaion. This may lead to a race condition which
    would result in an instance going into an error state as the volume
    operation is not atomic.

    Commit 4c4dc3a6d added the lock around the attach_volume. This
    follows that same pattern.

    Co-Authored-By: Matthew Booth <email address hidden>

    Change-Id: I7261135a44f0c2eca4732c8360b9c74b729991a8
    Closes-bug: #1662483

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/637534

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/637535

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/637536

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/rocky)

Reviewed: https://review.openstack.org/637534
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3310c3cbf534f5d75477ed206a8fb68eb53c6c10
Submitter: Zuul
Branch: stable/rocky

commit 3310c3cbf534f5d75477ed206a8fb68eb53c6c10
Author: Gary Kotton <email address hidden>
Date: Wed Feb 13 09:50:14 2019 -0800

    Lock detach_volume

    A detach_volume operation may take place during a attach_volume or
    delete_instance opertaion. This may lead to a race condition which
    would result in an instance going into an error state as the volume
    operation is not atomic.

    Commit 4c4dc3a6d added the lock around the attach_volume. This
    follows that same pattern.

    Co-Authored-By: Matthew Booth <email address hidden>

    Change-Id: I7261135a44f0c2eca4732c8360b9c74b729991a8
    Closes-bug: #1662483
    (cherry picked from commit 2e4e203c1a4c4f91dd1aabae8c8d7b745df99be7)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/queens)

Reviewed: https://review.openstack.org/637535
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=17b3d5734af9c7b916cfc7007e98f43df975b8e8
Submitter: Zuul
Branch: stable/queens

commit 17b3d5734af9c7b916cfc7007e98f43df975b8e8
Author: Gary Kotton <email address hidden>
Date: Wed Feb 13 09:50:14 2019 -0800

    Lock detach_volume

    A detach_volume operation may take place during a attach_volume or
    delete_instance opertaion. This may lead to a race condition which
    would result in an instance going into an error state as the volume
    operation is not atomic.

    Commit 4c4dc3a6d added the lock around the attach_volume. This
    follows that same pattern.

    Co-Authored-By: Matthew Booth <email address hidden>

    Change-Id: I7261135a44f0c2eca4732c8360b9c74b729991a8
    Closes-bug: #1662483
    (cherry picked from commit 2e4e203c1a4c4f91dd1aabae8c8d7b745df99be7)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/pike)

Reviewed: https://review.openstack.org/637536
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f0b833a352f8859c7b7e299d90440acfbd42e9db
Submitter: Zuul
Branch: stable/pike

commit f0b833a352f8859c7b7e299d90440acfbd42e9db
Author: Gary Kotton <email address hidden>
Date: Wed Feb 13 09:50:14 2019 -0800

    Lock detach_volume

    A detach_volume operation may take place during a attach_volume or
    delete_instance opertaion. This may lead to a race condition which
    would result in an instance going into an error state as the volume
    operation is not atomic.

    Commit 4c4dc3a6d added the lock around the attach_volume. This
    follows that same pattern.

    Co-Authored-By: Matthew Booth <email address hidden>

    Change-Id: I7261135a44f0c2eca4732c8360b9c74b729991a8
    Closes-bug: #1662483
    (cherry picked from commit 2e4e203c1a4c4f91dd1aabae8c8d7b745df99be7)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 19.0.0.0rc1

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 17.0.10

This issue was fixed in the openstack/nova 17.0.10 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 18.2.0

This issue was fixed in the openstack/nova 18.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 16.1.8

This issue was fixed in the openstack/nova 16.1.8 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.