OpenStack Compute (nova)

Failure in delete leads to undeletable instance.

Bug #1055095 reported by Vish Ishaya
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Vish Ishaya
OpenStack Compute (nova) 2012.2 "folsom"

Bug Description

If there is a failure in volume deletion during instance delete, then it is possible for an instance to get into a state where it can't be deleted.

Repro Example:

# create an instance with a delete on terminate volume
nova boot --image=<image-uuid> --flavor 1 --block_device_mapping vda=<volume-uuid>:::1 test
# snapshot the volume
nova volume-snapshot-create <volume-uuid> test-snapshot
# delete the instance
nova delete test

Result:

The instance is terminated but the db record sticks around because the volume cannot be deleted. Subsequent runs of delete do not help.

Expected:

The instance should be able to be terminated.

Revision history for this message
Vish Ishaya (vishvananda) wrote :

It is probably good to add some features to cinder to make it so volumes with snapshots can be deleted more easily, but in the meantime, we have already terminated the instance, so the best option is probably just to ignore the failure for the volume to be removed.

Changed in nova:
status: New → In Progress
importance: Undecided → High
assignee: nobody → Vish Ishaya (vishvananda)
milestone: none → folsom-rc2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/13542

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/13542
Committed: http://github.com/openstack/nova/commit/4a97a743c41e53d7278590e75913ff761ea319db
Submitter: Jenkins
Branch: master

commit 4a97a743c41e53d7278590e75913ff761ea319db
Author: Vishvananda Ishaya <email address hidden>
Date: Sun Sep 23 17:13:44 2012 +0000

    Allow deletion of instance with failed vol cleanup

    If something goes wrong when trying to delete volumes that have
    been marked delete_on_terminate, we leave an instance record in
    the database that can't be removed even though the instance is
    gone.

    Intelligent recovery at this point is very difficult, so for now
    just log a warning and let the instance be deleted. It is unfortunate
    that there isn't a good way to let the user know that his volume
    was not deleted, but this seems preferable to leaving an undeletable
    instance.

    A test is included to verify the behavior, as well as a note so that
    a better fix can be included later.

    Fixes bug 1055095

    Change-Id: I26e801fc3e9c56cf84d2b977295fcf580428879a

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (milestone-proposed)

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/13579

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (milestone-proposed)

Reviewed: https://review.openstack.org/13579
Committed: http://github.com/openstack/nova/commit/47241dd34caf44ca3b2281c1e6edce5d51763d76
Submitter: Jenkins
Branch: milestone-proposed

commit 47241dd34caf44ca3b2281c1e6edce5d51763d76
Author: Vishvananda Ishaya <email address hidden>
Date: Sun Sep 23 17:13:44 2012 +0000

    Allow deletion of instance with failed vol cleanup

    If something goes wrong when trying to delete volumes that have
    been marked delete_on_terminate, we leave an instance record in
    the database that can't be removed even though the instance is
    gone.

    Intelligent recovery at this point is very difficult, so for now
    just log a warning and let the instance be deleted. It is unfortunate
    that there isn't a good way to let the user know that his volume
    was not deleted, but this seems preferable to leaving an undeletable
    instance.

    A test is included to verify the behavior, as well as a note so that
    a better fix can be included later.

    Fixes bug 1055095

    Change-Id: I26e801fc3e9c56cf84d2b977295fcf580428879a

Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: folsom-rc2 → 2012.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.