OpenStack Compute (nova)

Xenserver: Failed power_off during "nova delete" leads to undeletable instance

Bug #1029132 reported by Kevin L. Mitchell on 2012-07-25

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Fix Released	Undecided	Kevin L. Mitchell	OpenStack Compute (nova) 2012.2 "folsom"

Bug Description

The "nova delete" call attempts to power_off the instance, which puts the instance into task_state powering_off. If the power_off fails, however (like, if the instance isn't running), deletion stops and the instance becomes undeletable. This could also affect rescues.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2012-07-25: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/10302

Changed in nova:
assignee:	nobody → Kevin L. Mitchell (klmitch)
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2012-07-25: Fix merged to nova (master)

Reviewed: https://review.openstack.org/10302
Committed: http://github.com/openstack/nova/commit/6f0fecbed3a72ce052662a6bc6441d6aac6b9d1f
Submitter: Jenkins
Branch: master

commit 6f0fecbed3a72ce052662a6bc6441d6aac6b9d1f
Author: Kevin L. Mitchell <email address hidden>
Date: Wed Jul 25 17:27:57 2012 -0500

Allow soft deletes from any state

    At some point in the past, the compute manager's delete() method had
    its @check_instance_state() decorator updated to allow deletes to be
    performed from any combination of vm_state and task_state. This same
    change was not made to the soft_delete() method, and so soft_delete()
    is only available if the task_state is clear, meaning that instances
    that are put into ERROR state with task_state set to something are
    effectively undeletable, if FLAGS.reclaim_instance_interval is set to
    some non-zero value (which is what causes the API DELETE call to use
    soft_delete() as opposed to delete()). This patch updates the
    @check_instance_state() on soft_delete() to match that on delete(),
    and fixes bug 1029130 and bug 1029133. This patch also partially
    fixes bug 1029132, but more needs to be done to completely fix that
    bug.

Change-Id: I78878110c696280f2bc28434433814884a526264

Changed in nova:
status:	In Progress → Fix Committed

Revision history for this message

Kevin L. Mitchell (klmitch) wrote on 2012-07-25:

Fix committed above only partially fixes the bug.

Changed in nova:
status:	Fix Committed → New

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2012-07-26: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/10366

Changed in nova:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2012-07-26: Fix merged to nova (master)

Reviewed: https://review.openstack.org/10366
Committed: http://github.com/openstack/nova/commit/27826a84cc603772dd4fb9fc7360b27e0f52b322
Submitter: Jenkins
Branch: master

commit 27826a84cc603772dd4fb9fc7360b27e0f52b322
Author: Kevin L. Mitchell <email address hidden>
Date: Thu Jul 26 13:49:09 2012 -0500

Allow power_off when instance doesn't exist

    Certain errors that can occur while booting an instance may leave no
    VM on the Xen hypervisor. This can make that instance undeletable,
    since the first step in soft_delete() is to power_off the instance,
    and nothing catches the NotFound error thrown if the instance doesn't
    exist. The libvirt support handles the case of a non-existant
    instance by ignoring NotFound--after all, a non-existant instance is,
    by definition, powered off, right?

Fixes bug 1029132.

Change-Id: I5bffe3b61fe92fc32fb70cab941fa5ecd250df77