Fuel for OpenStack

TransactionRollbackError: deadlock detected

Bug #1340703 reported by Anastasia Palkina
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Alexander Kislitsky
Fuel for OpenStack 5.1
5.0.x
Won't Fix
High
Alexander Kislitsky
Fuel for OpenStack 5.0.2

Bug Description

"build_id": "2014-07-10_00-39-56",
"mirantis": "yes",
"build_number": "112",
"ostf_sha": "09b6bccf7d476771ac859bb3c76c9ebec9da9e1f",
"nailgun_sha": "f5ff82558f99bb6ca7d5e1617eddddf7142fe857",
"production": "docker",
"api": "1.0",
"fuelmain_sha": "293015843304222ead899270449495af91b06aed",
"astute_sha": "5df009e8eab611750309a4c5b5c9b0f7b9d85806",
"release": "5.0.1",
"fuellib_sha": "364dee37435cbdc85d6b814a61f57800b83bf22d"

There is errors in RPC Consumer log.

2014-07-10 11:44:38 ERROR

[7f82b783c700] (receiverd) Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/nailgun/rpc/receiverd.py", line 47, in consume_msg
    callback(**body["args"])
  File "/usr/lib/python2.6/site-packages/nailgun/rpc/receiver.py", line 190, in deploy_resp
    [n['uid'] for n in nodes]
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2227, in __iter__
    return self._execute_and_instances(context)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1449, in execute
    params)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1717, in _execute_context
    result = context.get_result_proxy()
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 246, in get_result_proxy
    return base.BufferedRowResultProxy(self)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 2903, in __init__
    self._init_metadata()
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 3306, in _init_metadata
    self.__buffer_rows()
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 3326, in __buffer_rows
    self.__rowbuffer = collections.deque(self.cursor.fetchmany(size))
TransactionRollbackError: deadlock detected
DETAIL: Process 699 waits for ShareLock on transaction 3615; blocked by process 717.
Process 717 waits for ShareLock on transaction 3614; blocked by process 699.
HINT: See server log for query details.

Logs are here: https://drive.google.com/a/mirantis.com/file/d/0B6SjzarTGFxaMEgxNmozajJUbUE/edit?usp=sharing

Mike Scherbakov (mihgen)
Changed in fuel:
milestone: 5.0.2 → 5.0.1
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Alexander Kislitsky (akislitsky)
status: New → In Progress
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
milestone: 5.0.1 → 5.1
no longer affects: fuel/5.1.x
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (master)

Fix proposed to branch: master
Review: https://review.openstack.org/107065

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (master)

Reviewed: https://review.openstack.org/107065
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=8b3d076e4f2cd6bafd0bed978dd405affaa8f2d1
Submitter: Jenkins
Branch: master

commit 8b3d076e4f2cd6bafd0bed978dd405affaa8f2d1
Author: Alexander Kislitsky <email address hidden>
Date: Tue Jul 15 18:08:10 2014 +0400

    Deadlocks and random CI tests failures are fixed

    Code partially refactored to proper use of DB transactions:
     - receiver partially refactored from DB models to Nailgun objects
     - locking of objects added into reveiver
     - task manager partially refactored
     - DB commits removed from receiver
     - weird DB commit removed from Nailgun task object
     - weird logic for repeating tests on deadlocks removed
     - in fake tasks reveiver methods calling wrapped by try/except
     - filtering of empty collection in NailgunCollection fixed
     - ordering by id added into getting Task object by uuid in case of locking
     - values from consts module used instead hardcoded values in many places

    Implements blueprint nailgun-db-transactions
    Closes-Bug: #1340703

    Change-Id: I520efc9a4b0b52feae4b86309da653ade50c7c9c

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Alexander Kislitsky (akislitsky) wrote :

This fix should be tested by QA team before backporting to 5.0.x

Revision history for this message
Anastasia Palkina (apalkina) wrote :

Verified on latests ISOs
"build_id": "2014-08-05_02-01-14",
"ostf_sha": "a3fa823ea0e4e03beb637ae07a91adea82c33182",
"build_number": "393",
"auth_required": true,
"api": "1.0",
"nailgun_sha": "5d1e8e38ea3f2cd88ceca8e0927d53e9770598bc",
"production": "docker",
"fuelmain_sha": "6b05280621c23ab158e9943f88b6f71a22fd4c69",
"astute_sha": "99a790ad1b7526cbbd5bf8add0cb2b4e503fccd4",
"feature_groups": ["mirantis"],
"release": "5.1",
"fuellib_sha": "25eb632c70fe0bb3b2c1053314b558a160e4af50"

Changed in fuel:
status: Fix Committed → Fix Released
Revision history for this message
Dmitry Pyzhov (dpyzhov) wrote :

We do not support master node for 5.0.2 release

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.