Just saw it again in the gate here:
http://logs.openstack.org/40/627540/8/gate/tempest-full-py3/b25fd48/controller/logs/screen-n-api.txt.gz?level=TRACE#_Jan_24_19_53_01_166399
Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi pymysql.err.OperationalError: (1213, 'Deadlock found when trying to get lock; try restarting transaction') Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi The above exception was the direct cause of the following exception: Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi Traceback (most recent call last): Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 671, in wrapped Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi return f(*args, **kwargs) Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 763, in _action_revert_resize Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi self.compute_api.revert_resize(context, instance) Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/api.py", line 206, in inner Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi return function(self, context, instance, *args, **kwargs) Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/api.py", line 214, in _wrapped Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi return fn(self, context, instance, *args, **kwargs) Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/api.py", line 154, in inner Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi return f(self, context, instance, *args, **kw) Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/api.py", line 3455, in revert_resize Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi migration.save() Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python3.6/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi return fn(self, *args, **kwargs) Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/objects/migration.py", line 164, in save Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi db_migration = db.migration_update(self._context, self.id, updates) Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/db/api.py", line 518, in migration_update Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi return IMPL.migration_update(context, id, values) Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 227, in wrapped Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi return f(context, *args, **kwargs) Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi next(self.gen) Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python3.6/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1062, in _transaction_scope Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi yield resource Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ Jan 24 19:53:01.166399 ubuntu-bionic-ovh-bhs1-0002105626 <email address hidden>[9264]: ERROR nova.api.openstack.wsgi next(self.gen)
We should probably put a retry_on_deadlock decorator on the migration_update method.
Just saw it again in the gate here:
http:// logs.openstack. org/40/ 627540/ 8/gate/ tempest- full-py3/ b25fd48/ controller/ logs/screen- n-api.txt. gz?level= TRACE#_ Jan_24_ 19_53_01_ 166399
Jan 24 19:53:01.166399 ubuntu- bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi pymysql. err.Operational Error: (1213, 'Deadlock found when trying to get lock; try restarting transaction') bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi The above exception was the direct cause of the following exception: bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi Traceback (most recent call last): bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi File "/opt/stack/ nova/nova/ api/openstack/ wsgi.py" , line 671, in wrapped bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi return f(*args, **kwargs) bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi File "/opt/stack/ nova/nova/ api/openstack/ compute/ servers. py", line 763, in _action_ revert_ resize bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi self.compute_ api.revert_ resize( context, instance) bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi File "/opt/stack/ nova/nova/ compute/ api.py" , line 206, in inner bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi return function(self, context, instance, *args, **kwargs) bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi File "/opt/stack/ nova/nova/ compute/ api.py" , line 214, in _wrapped bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi return fn(self, context, instance, *args, **kwargs) bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi File "/opt/stack/ nova/nova/ compute/ api.py" , line 154, in inner bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi return f(self, context, instance, *args, **kw) bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi File "/opt/stack/ nova/nova/ compute/ api.py" , line 3455, in revert_resize bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi migration.save() bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi File "/usr/local/ lib/python3. 6/dist- packages/ oslo_versionedo bjects/ base.py" , line 226, in wrapper bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi return fn(self, *args, **kwargs) bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi File "/opt/stack/ nova/nova/ objects/ migration. py", line 164, in save bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi db_migration = db.migration_ update( self._context, self.id, updates) bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi File "/opt/stack/ nova/nova/ db/api. py", line 518, in migration_update bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi return IMPL.migration_ update( context, id, values) bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi File "/opt/stack/ nova/nova/ db/sqlalchemy/ api.py" , line 227, in wrapped bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi return f(context, *args, **kwargs) bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi File "/usr/lib/ python3. 6/contextlib. py", line 88, in __exit__ bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi next(self.gen) bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi File "/usr/local/ lib/python3. 6/dist- packages/ oslo_db/ sqlalchemy/ enginefacade. py", line 1062, in _transaction_scope bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi yield resource bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi File "/usr/lib/ python3. 6/contextlib. py", line 88, in __exit__ bionic- ovh-bhs1- 0002105626 <email address hidden>[9264]: ERROR nova.api. openstack. wsgi next(self.gen)
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
Jan 24 19:53:01.166399 ubuntu-
We should probably put a retry_on_deadlock decorator on the migration_update method.