Async tasks (cluster deletion, resetting, snapshot generation) sometimes fail

The root cause of the issue is that having multiple threads and using scoped_session and not correctly synchronizing those threads could lead to database locks and block these threads. In nailgun tests we could mock rpc and check arguments that are used when rpc is called.

When using fake thread for UI tests this mock approach can not be used. Looks like we need to correctly synchronize application threads.

It seems that the fails rate (due to shared database locks) depends on the test node performance. For instance, the same patch could pass tests on openstack-infra while it likely fails when running on slow VM. (there is no enough data to to say this is true, it is just an observation).

increase priority to Critical, because: all tests are falling and merge of code was blocked

I see next options to fix:

1. fix failures on shared database locks. they can be introduced by new features or refactoring.
2. remove fake threads.

@Roman, could you please provide links to tests failures. we need more information to find out and fix the issues.

@Alex,

It's not an option to remove fake threads. Those failures are about UI integration tests, not about nailgun's one.

Related fix proposed to branch: master
Review: https://review.openstack.org/291206

Related fix proposed to branch: mos/mitaka
Change author: Vitaly Kramskikh <email address hidden>
Review: https://review.fuel-infra.org/17918

Reviewed: https://review.openstack.org/291206
Committed: https://git.openstack.org/cgit/openstack/fuel-ui/commit/?id=4a00826795be919f388d22a387d5aa512801a5cd
Submitter: Jenkins
Branch: master

commit 4a00826795be919f388d22a387d5aa512801a5cd
Author: Vitaly Kramskikh <email address hidden>
Date: Thu Mar 10 17:31:17 2016 +0300

    Avoid cluster deletion when testing Healtcheck tab

    Fake cluster deletion for some reason fails very often due
    to a bug, so it's better to avoid it. It's not needed at all to
    test Healthcheck tab.

    Change-Id: I5a1a010d819b75b7ae8a8be9d9bf9d359c3d191b
    Related-Bug: #1549750

Change abandoned by Vitaly Kramskikh <email address hidden> on branch: mos/mitaka
Review: https://review.fuel-infra.org/17918

Change abandoned by Alexander Kislitsky (<email address hidden>) on branch: master
Review: https://review.openstack.org/291245
Reason: This cosmetic change is not required

Related fix proposed to branch: master
Review: https://review.openstack.org/291749

Reviewed: https://review.openstack.org/291749
Committed: https://git.openstack.org/cgit/openstack/fuel-ui/commit/?id=e506310ea0fa065d62b207b73dbc7adab231f48f
Submitter: Jenkins
Branch: master

commit e506310ea0fa065d62b207b73dbc7adab231f48f
Author: Vitaly Kramskikh <email address hidden>
Date: Fri Mar 11 16:56:13 2016 +0300

    Fix random test failures for specific browser/screen combinations

    Related-Bug: #1549750

    Change-Id: Ia1a9060b8d7d490db35f817598b26761e290a3b2

Change abandoned by Vitaly Kramskikh <email address hidden> on branch: mos/mitaka
Review: https://review.fuel-infra.org/17983

Related fix proposed to branch: master
Review: https://review.openstack.org/292856

Reviewed: https://review.openstack.org/292856
Committed: https://git.openstack.org/cgit/openstack/fuel-ui/commit/?id=102bf656d0c8e64c7b80d9ac76373d0460ce610e
Submitter: Jenkins
Branch: master

commit 102bf656d0c8e64c7b80d9ac76373d0460ce610e
Author: Vitaly Kramskikh <email address hidden>
Date: Tue Mar 15 15:27:46 2016 +0300

    Rewrite UI plugin tests to avoid cluster creation/deletion

    There are 3 consecutive deployment/deletion calls which
    often cause failure on fake environment due to a bug in
    Nailgun. This test case was rewritten to use only 1 deployment
    call.

    Change-Id: Idd43240855d7e6b538b67abec61bd2c0e6d90bab
    Related-Bug: #1549750

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

version

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

Related fix proposed to branch: master
Review: https://review.openstack.org/295881

Reviewed: https://review.openstack.org/295881
Committed: https://git.openstack.org/cgit/openstack/fuel-ui/commit/?id=d81041ca9125a2797bd5b436b2d8950dfeeed47c
Submitter: Jenkins
Branch: master

commit d81041ca9125a2797bd5b436b2d8950dfeeed47c
Author: Kate Pimenova <email address hidden>
Date: Tue Mar 22 17:57:22 2016 +0300

    Fix for random ui test fail

    Change-Id: I22e1cf620c946829a118cb15a0972b5c6c6d7c66
    Related-Bug: #1549750

Per Vitaly's comment in https://review.openstack.org/#/c/294976/, patchset doesn't deal with this bug. Removing myself as assignee.

Technical debt, doesn't affect real users. Removing from Mitaka release.

This bug doesn't block development