Comment 4 for bug 1642537

Revision history for this message
Matt Riedemann (mriedem) wrote :

Just saw it again in the gate here:

http://logs.openstack.org/40/627540/8/gate/tempest-full-py3/b25fd48/controller/logs/screen-n-api.txt.gz?level=TRACE#_Jan_24_19_53_01_166399

Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi pymysql.err.OperationalError: (1213, 'Deadlock found when trying to get lock; try restarting transaction')
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi The above exception was the direct cause of the following exception:
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi Traceback (most recent call last):
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 671, in wrapped
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi return f(*args, **kwargs)
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 763, in _action_revert_resize
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi self.compute_api.revert_resize(context, instance)
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/api.py", line 206, in inner
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi return function(self, context, instance, *args, **kwargs)
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/api.py", line 214, in _wrapped
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi return fn(self, context, instance, *args, **kwargs)
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/api.py", line 154, in inner
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi return f(self, context, instance, *args, **kw)
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/api.py", line 3455, in revert_resize
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi migration.save()
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python3.6/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi return fn(self, *args, **kwargs)
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/objects/migration.py", line 164, in save
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi db_migration = db.migration_update(self._context, self.id, updates)
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/db/api.py", line 518, in migration_update
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi return IMPL.migration_update(context, id, values)
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 227, in wrapped
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi return f(context, *args, **kwargs)
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi next(self.gen)
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python3.6/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1062, in _transaction_scope
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi yield resource
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi next(self.gen)

We should probably put a retry_on_deadlock decorator on the migration_update method.