Fuel for OpenStack

Provisioning randomly failed during web ui func tests

Bug #1615557 reported by Julia Aranovich
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
Critical
Vitaly Kramskikh
Fuel for OpenStack 10.0
Mitaka
Fix Committed
Critical
Vitaly Kramskikh
Fuel for OpenStack 9.1

Bug Description

Web UI func tests frequently fail with the following error:
https://ci.fuel-infra.org/job/verify-fuel-ui-on-fuel-web/4179/console

"AssertionError: Provisioning finished: expected 1 to equal 0"

Looks like there is a problem with tasks on backend (fake tasks?):
https://ci.fuel-infra.org/job/verify-fuel-ui-on-fuel-web/4179/artifact/artifacts/nailgun.log

ObjectNotFound: Task with UUID=37baa29e-db5b-4543-a855-837f542e746c is not found in DB

Julia Aranovich (jkirnosova)
summary: - Random failure of web ui func tests
+ Provisioning randomly failed during web ui func tests
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

This issue is blocker for commits on review.

Please take a look and fix it as soon as possible.

Thank you!

Changed in fuel:
importance: High → Critical
Revision history for this message
Alexander Gordeev (a-gordeev) wrote :

unless it's fixed, few fuel:recheck could help to make the job green again

Georgy Kibardin (gkibardin)
Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Georgy Kibardin (gkibardin)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-web (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/359956

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (master)

Fix proposed to branch: master
Review: https://review.openstack.org/360066

Changed in fuel:
assignee: Georgy Kibardin (gkibardin) → Bulat Gaifullin (bgaifullin)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-ui (master)

Reviewed: https://review.openstack.org/360019
Committed: https://git.openstack.org/cgit/openstack/fuel-ui/commit/?id=4dc1c4476e33f4189f81a9fc1fabd69eb1995578
Submitter: Jenkins
Branch: master

commit 4dc1c4476e33f4189f81a9fc1fabd69eb1995578
Author: Dmitry Guryanov <email address hidden>
Date: Wed Aug 24 20:20:59 2016 +0300

    Disable provision and deployment tests temporarily

    These tests often fail due to Nailgun bugs

    Related-Bug: #1615557

    Change-Id: Ia332506e1716b29866ed143198748651cedaf3fc

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-web (master)

Change abandoned by Georgy Kibardin (<email address hidden>) on branch: master
Review: https://review.openstack.org/359956

Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

Do we plan to prepare another fix?

Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

We don't have a fix on review for stable/mitaka, status changed to Confirmed for 9.1

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/362913

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (master)

Reviewed: https://review.openstack.org/360066
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=03acc6603f710f7d6a820e55154298b33f5abc11
Submitter: Jenkins
Branch: master

commit 03acc6603f710f7d6a820e55154298b33f5abc11
Author: Bulat Gaifullin <email address hidden>
Date: Wed Aug 24 23:11:09 2016 +0300

    Do not allow to run 2 or more tasks simultaneously

    Simultaneous run of 2 or tasks may cause
    side-effects and the simplest way is prevent this action.
    Also fixed all places where rpc.cast called without commit

    Change-Id: I029768900d345540c3b501f1fa3649b063d3a55d
    Partial-Bug: 1615557

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (master)

Fix proposed to branch: master
Review: https://review.openstack.org/362985

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (master)

Reviewed: https://review.openstack.org/362985
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=f01af6274f9c24d7e76c8e6922e5a06196c92047
Submitter: Jenkins
Branch: master

commit f01af6274f9c24d7e76c8e6922e5a06196c92047
Author: Bulat Gaifullin <email address hidden>
Date: Tue Aug 30 18:55:13 2016 +0300

    Fixed dead-lock durring deletion of obsolete task

    Also fixed raice condition in deploy_resp

    Change-Id: I9eeff20ac0c052b2bab8e888fe8420af2f49bbe4
    Partial-Bug: 1615557

Alexey Shtokolov (ashtokolov)
tags: added: non-release
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (stable/mitaka)

Reviewed: https://review.openstack.org/362913
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=cb36f4a7b7b2e4806c3952f5b367f75cd8bdd6fa
Submitter: Jenkins
Branch: stable/mitaka

commit cb36f4a7b7b2e4806c3952f5b367f75cd8bdd6fa
Author: Bulat Gaifullin <email address hidden>
Date: Wed Aug 24 23:11:09 2016 +0300

    Do not allow to run 2 or more tasks simultaneously

    Simultaneous run of 2 or tasks may cause
    side-effects and the simplest way is prevent this action.
    Also fixed all places where rpc.cast called without commit

    Change-Id: I029768900d345540c3b501f1fa3649b063d3a55d
    Partial-Bug: 1615557

tags: added: in-stable-mitaka
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-ui (master)

Fix proposed to branch: master
Review: https://review.openstack.org/364256

Changed in fuel:
assignee: Bulat Gaifullin (bgaifullin) → Vitaly Kramskikh (vkramskikh)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-ui (master)

Reviewed: https://review.openstack.org/364256
Committed: https://git.openstack.org/cgit/openstack/fuel-ui/commit/?id=963eeb6e3b9513fe4f04d11788c7f33f4b987cee
Submitter: Jenkins
Branch: master

commit 963eeb6e3b9513fe4f04d11788c7f33f4b987cee
Author: Vitaly Kramskikh <email address hidden>
Date: Thu Sep 1 14:58:38 2016 +0300

    Set all provisioning/deployment timeouts in UI func tests to 60s

    Current ones may not be sufficient and cause random failures.

    Change-Id: I061e8281812db388f23797b33662ea401a93679b
    Partial-Bug: #1615557

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-ui (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/364308

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-ui (stable/mitaka)

Reviewed: https://review.openstack.org/364308
Committed: https://git.openstack.org/cgit/openstack/fuel-ui/commit/?id=c59dacc46812f11523d11b3cc69a8559c31011e2
Submitter: Jenkins
Branch: stable/mitaka

commit c59dacc46812f11523d11b3cc69a8559c31011e2
Author: Vitaly Kramskikh <email address hidden>
Date: Thu Sep 1 16:25:39 2016 +0300

    Set all provisioning/deployment timeouts in UI func tests to 60s

    Current ones may not be sufficient and cause random failures.

    Change-Id: I061e8281812db388f23797b33662ea401a93679b
    Partial-Bug: #1615557

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-ui (master)

Fix proposed to branch: master
Review: https://review.openstack.org/365693

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-ui (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/365712

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-ui (master)

Reviewed: https://review.openstack.org/365693
Committed: https://git.openstack.org/cgit/openstack/fuel-ui/commit/?id=21ec29a4ee42ab764ca35713280414b020944919
Submitter: Jenkins
Branch: master

commit 21ec29a4ee42ab764ca35713280414b020944919
Author: Vitaly Kramskikh <email address hidden>
Date: Mon Sep 5 17:58:54 2016 +0300

    Increase notification tests timeouts

    Current timeouts may not be enough in some cases, so this
    commit increases them to 30 seconds.

    Change-Id: I31f45e5d59ac015c1544f3bd0099a49d7a9b5b17
    Partial-Bug: #1615557

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-ui (stable/mitaka)

Reviewed: https://review.openstack.org/365712
Committed: https://git.openstack.org/cgit/openstack/fuel-ui/commit/?id=0c669409dc70e162dbf761750d8a0052883570d3
Submitter: Jenkins
Branch: stable/mitaka

commit 0c669409dc70e162dbf761750d8a0052883570d3
Author: Vitaly Kramskikh <email address hidden>
Date: Mon Sep 5 17:58:54 2016 +0300

    Increase notification tests timeouts

    Current timeouts may not be enough in some cases, so this
    commit increases them to 30 seconds.

    Change-Id: I31f45e5d59ac015c1544f3bd0099a49d7a9b5b17
    Partial-Bug: #1615557

Revision history for this message
Vitaly Kramskikh (vkramskikh) wrote :

It seems latest patches to fuel-web and fuel-ui repos fixed this issue or at least significantly reduces random failure rate. So this issue can be considered as fixed.

Changed in fuel:
status: In Progress → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.