OpenStack Compute (nova)

Comment 1 for bug 1642537

Revision history for this message

Matt Riedemann (mriedem) wrote on 2016-12-07: Re: Request fails with DBDeadlock

2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [req-027e93fd-83f6-4c9a-8e42-bc3544a52402 tempest-ServerActionsTestJSON-1155389376 tempest-ServerActionsTestJSON-1155389376] [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565] Setting instance vm_state to ERROR
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565] Traceback (most recent call last):
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565] File "/opt/stack/new/nova/nova/compute/manager.py", line 3963, in finish_resize
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565] disk_info, image_meta)
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565] File "/opt/stack/new/nova/nova/compute/manager.py", line 3932, in _finish_resize
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565] migration.save()
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565] File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 210, in wrapper
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565] ctxt, self, fn.__name__, args, kwargs)
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565] File "/opt/stack/new/nova/nova/conductor/rpcapi.py", line 241, in object_action
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565] objmethod=objmethod, args=args, kwargs=kwargs)
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565] retry=self.retry)
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 97, in _send
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565] timeout=timeout, retry=retry)
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 467, in send
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565] retry=retry)
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 458, in _send
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565] raise result
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565] RemoteError: Remote error: DBDeadlock (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'UPDATE migrations SET updated_at=%(updated_at)s, status=%(status)s WHERE migrations.id = %(migrations_id)s'] [parameters: {'status': u'finished', 'migrations_id': 6, 'updated_at': datetime.datetime(2016, 11, 17, 4, 43, 55, 648839)}]

2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [req-027e93fd-83f6-4c9a-8e42-bc3544a52402 tempest-ServerActionsTestJSON-1155389376 tempest-ServerActionsTestJSON-1155389376] [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565] Setting instance vm_state to ERROR
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565] Traceback (most recent call last):
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565]   File "/opt/stack/new/nova/nova/compute/manager.py", line 3963, in finish_resize
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565]     disk_info, image_meta)
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565]   File "/opt/stack/new/nova/nova/compute/manager.py", line 3932, in _finish_resize
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565]     migration.save()
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565]   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 210, in wrapper
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565]     ctxt, self, fn.__name__, args, kwargs)
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565]   File "/opt/stack/new/nova/nova/conductor/rpcapi.py", line 241, in object_action
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565]     objmethod=objmethod, args=args, kwargs=kwargs)
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565]     retry=self.retry)
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 97, in _send
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565]     timeout=timeout, retry=retry)
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 467, in send
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565]     retry=retry)
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 458, in _send
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565]     raise result
2016-11-17 04:43:55.673 3598 ERROR nova.compute.manager [instance: 00876f9f-2f30-4507-93c0-f0f5a1699565] RemoteError: Remote error: DBDeadlock (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'UPDATE migrations SET updated_at=%(updated_at)s, status=%(status)s WHERE migrations.id = %(migrations_id)s'] [parameters: {'status': u'finished', 'migrations_id': 6, 'updated_at': datetime.datetime(2016, 11, 17, 4, 43, 55, 648839)}]