Async tasks (cluster deletion, resetting, snapshot generation) sometimes fail
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Confirmed
|
High
|
Fuel Sustaining | ||
Mitaka |
Won't Fix
|
High
|
Fuel Python (Deprecated) |
Bug Description
Not sure if it affects our normal deployment, it affects at least our UI func tests.
Some examples:
https:/
https:/
In app.log from artifacts you can see tracebacks like
(cluster deletion)
Exception in thread EXECUTE_TASKS:
Traceback (most recent call last):
File "/usr/lib/
self.run()
File "/home/
self.
File "/home/
resp_
File "/home/
objects.
File "/home/
logger.
AttributeError: 'NoneType' object has no attribute 'uuid'
or
(diagnostic snapshot generation)
2016-02-25 10:10:47.434 ERROR [7fe4eb25e700] (manager) Traceback (most recent call last):
File "/home/
to_return = method(task, *args, **kwargs)
File "/home/
'settings': conf or cls.conf()
File "/home/
editable_attrs = node.cluster.
AttributeError: 'NoneType' object has no attribute 'attributes'
description: | updated |
summary: |
- Cluster deletion sometimes fails + Async tasks (cluster deletion, resetting, snapshot generation) sometimes + fail |
tags: | added: tech-debt |
Changed in fuel: | |
assignee: | Fuel Python Team (fuel-python) → Alexander Kislitsky (akislitsky) |
Changed in fuel: | |
status: | In Progress → Confirmed |
Changed in fuel: | |
assignee: | Alexander Kislitsky (akislitsky) → Fuel Python Team (fuel-python) |
Changed in fuel: | |
assignee: | Fuel Python Team (fuel-python) → Mike Scherbakov (mihgen) |
status: | Confirmed → In Progress |
Changed in fuel: | |
assignee: | Fuel UI Team (fuel-ui) → Fuel Python Team (fuel-python) |
Changed in fuel: | |
status: | In Progress → Confirmed |
tags: | removed: need-info |
Changed in fuel: | |
milestone: | 9.0 → 10.0 |
Changed in fuel: | |
assignee: | Fuel Python (Deprecated) (fuel-python) → Fuel Sustaining (fuel-sustaining-team) |
The root cause of the issue is that having multiple threads and using scoped_session and not correctly synchronizing those threads could lead to database locks and block these threads. In nailgun tests we could mock rpc and check arguments that are used when rpc is called.
When using fake thread for UI tests this mock approach can not be used. Looks like we need to correctly synchronize application threads.
It seems that the fails rate (due to shared database locks) depends on the test node performance. For instance, the same patch could pass tests on openstack-infra while it likely fails when running on slow VM. (there is no enough data to to say this is true, it is just an observation).