Description
===========
I use "nova resize" command to resize the vm ,but I don't call "nova resize-confirm" ,then I use "nova delete" command to soft-delete the vm ,it reports ERROR as this:
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'UnexpectedTaskStateError_Remote'> (HTTP 500) (Request-ID: req-2ee1cacd-78d8-4a9d-b975-dec7d31c8033)
ERROR (CommandError): Unable to delete the specified server(s).
Steps to reproduce
==================
1)I config nova.conf, make reclaim_instance_interval equal to 60 to enable "soft-delete",
then restart openstack-nova-api service, restart openstack-nova-compute service.
1)resize vm1 by "nova resize" command line
2)delete vm1 by "nova delete" command line
Environment
===========
The branch of OpenStack that I use is "mitaka"
Expected result
===============
The vm is deleted
Actual result
=============
The vm is not deleted ,and report ERROR.
Logs & Configs
==============
nova.conf:
reclaim_instance_interval = 60
nova-api.log:
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions [req-2ee1cacd-78d8-4a9d-b975-dec7d31c8033 fbc7acf5e66e4a2289f419d257ffba84 95579f6a417e41ca8e8a74fdd8b23b14 - - -] Unexpected exception in API method
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 959, in delete
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 809, in _delete
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 170, in wrapped
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 159, in inner
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 187, in _wrapped
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 140, in inner
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1928, in soft_delete
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1708, in _delete
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions self.force_reraise()
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb)
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1661, in _delete
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1757, in _confirm_resize_on_deleting
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 500, in confirm_resize
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions retry=self.retry)
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions timeout=timeout, retry=retry)
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions retry=retry)
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 461, in _send
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions raise result
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions UnexpectedTaskStateError_Remote: Conflict updating instance 282905b9-e342-4a1f-8b13-90750552d8de. Expected: {'task_state': [None, u'deleting']}. Actual: {'task_state': u'soft-deleting'}
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 90, in _object_dispatch
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 223, in wrapper
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions return fn(self, *args, **kwargs)
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 701, in save
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 809, in instance_update_and_get_original
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 229, in wrapper
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 148, in wrapper
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions ectxt.value = e.inner_exc
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions self.force_reraise()
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb)
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 300, in wrapped
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 2717, in instance_update_and_get_original
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 2853, in _instance_update
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions UnexpectedTaskStateError: Conflict updating instance 282905b9-e342-4a1f-8b13-90750552d8de. Expected: {'task_state': [None, u'deleting']}. Actual: {'task_state': u'soft-deleting'}
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions incoming.message))
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions return self._do_dispatch(endpoint, method, ctxt, args)
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions result = func(ctxt, **new_args)
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/exception.py", line 110, in wrapped
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions self.force_reraise()
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb)
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 414, in decorated_function
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions return function(self, context, *args, **kwargs)
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 392, in decorated_function
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions kwargs['instance'], e, sys.exc_info())
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions self.force_reraise()
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb)
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 380, in decorated_function
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions return function(self, context, *args, **kwargs)
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3529, in confirm_resize
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions do_confirm_resize(context, instance, migration.id)
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3527, in do_confirm_resize
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions migration=migration)
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3581, in _confirm_resize
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions instance.save(expected_task_state=[None, task_states.DELETING])
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 207, in wrapper
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions ctxt, self, fn.__name__, args, kwargs)
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 245, in object_action
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions retry=self.retry)
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions timeout=timeout, retry=retry)
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions retry=retry)
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 461, in _send
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions raise result
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions UnexpectedTaskStateError_Remote: Conflict updating instance 282905b9-e342-4a1f-8b13-90750552d8de. Expected: {'task_state': [None, u'deleting']}. Actual: {'task_state': u'soft-deleting'}
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 90, in _object_dispatch
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 223, in wrapper
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions return fn(self, *args, **kwargs)
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 701, in save
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 809, in instance_update_and_get_original
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 229, in wrapper
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 148, in wrapper
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions ectxt.value = e.inner_exc
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions self.force_reraise()
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb)
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 300, in wrapped
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 2717, in instance_update_and_get_original
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 2853, in _instance_update
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions UnexpectedTaskStateError: Conflict updating instance 282905b9-e342-4a1f-8b13-90750552d8de. Expected: {'task_state': [None, u'deleting']}. Actual: {'task_state': u'soft-deleting'}
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
2017-08-23 10:45:38.233 1458420 ERROR nova.api.openstack.extensions
Hi,
I am able to reproduce this issue on current master as well.
Nova master: 9f3520672facff9 e7e25aa60b
commit f28da08a1b26b78
Merge: 6824847 5477259
Author: Jenkins <email address hidden>
Date: Thu Aug 17 04:55:56 2017 +0000
Merge "Fix contributor documentation"