Cannot delete vm instance if send duplicate delete requests

Bug #1431860 reported by Abhishek Kekane
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Confirmed
Low
Unassigned

Bug Description

I deployed openstack with icehouse rc1 and booted 100 vms on my nodes. After my testing, i tried to delete my vms at the same time. Then i fount all of my vms` status change to deleting but cannot be deleted. I checked my openstack, the rabbitmq-server crashed . Then i restart rabbitmq-server and my openstack nova services, sended the delete requests again and again, the vms still cannot be deleted. While , in havana, the vms can be deleted if received duplicate delete requests .
I think icehouse should handle duplicate delete requests like havana .

Note:
This bug is already reported in launchpad [1] but the fix [2] proposed to resolve it was reverted back as it was breaking the cells.

[1] https://bugs.launchpad.net/nova/+bug/1308342
[2] https://review.openstack.org/121800

Changed in nova:
assignee: nobody → Abhishek Kekane (abhishek-kekane)
status: New → In Progress
Changed in nova:
importance: Undecided → Low
Revision history for this message
Abhishek Kekane (abhishek-kekane) wrote :

Hi,

In Normal case, user will try to delete instance using command [1]. If user is not able to delete the instance
using command [1], then only he/she will try to force-delete it using command [2].

IMO the sole purpose of 'force-delete' is to delete an instance if it gets stuck in unrecoverable state.

In current bug, instance gets stuck in deleting task_state and since user can't reset the instance state, there is no way
user can delete that instance.
The proposed patch has a side effect, if user tries to force-delete the same instance multiple times. In that case,
nova-api server will be sending lots of rabbit messages to compute node to force-delete single instance.

Can we add new task_state 'force_deleting' which is set if instance is deleted using force-delete command [2] ?
If yes, we can add a check to reject subsequent force-delete requests.

[1] $ nova delete <instance_id>
[2] $ nova force-delete <instance_id>

Please give your opinion for the same.

Changed in nova:
assignee: Abhishek Kekane (abhishek-kekane) → Rajesh Tailor (rajesh-tailor)
melanie witt (melwitt)
tags: added: compute
removed: ntt
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Rajesh Tailor (<email address hidden>) on branch: master
Review: https://review.openstack.org/163798
Reason: The following scenario will not be fulfilled by proposed changes and issue will remain with newly introduced task_state ('force-deleting') as well, so abandoning this change.

When user sends force-delete request for first time for instance deletion, its task_state gets changed to 'force-deleting'. Now since the instance task_state is already in 'force-deleting' state and if at that moment rabbitmq-server get crashed by some reasons, then the instance task_state remain 'force-deleting' forever and instance will not be deleted.

Now after rabbitmq-server is up and user request for instance deletion using force-delete or delete api, it will raise InstanceInvalidState Exception and user won't be able to delete that instance ever without restarting nova-compute service.

Changed in nova:
assignee: Rajesh Tailor (rajesh-tailor) → nobody
status: In Progress → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Michael Still (<email address hidden>) on branch: master
Review: https://review.openstack.org/163798
Reason: This patch has been idle for a long time, so I am abandoning it to keep the review clean sane. If you're interested in still working on this patch, then please unabandon it and upload a new patchset.

Revision history for this message
Sean Dague (sdague) wrote :

Automatically discovered version icehouse in description. If this is incorrect, please update the description to include 'nova version: ...'

tags: added: openstack-version.icehouse
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.