Fuel for OpenStack

"Cobbler error" appears if two clusters are started to deploy in the same moment

Bug #1396181 reported by Dennis Dmitriev
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Vladimir Sharshov
Fuel for OpenStack 6.0
5.1.x
Fix Committed
High
Vladimir Sharshov
Fuel for OpenStack 5.1.1

Bug Description

Steps to reproduce:
    1. Create two clusters with 1 controller and 1 compute in each,
    2. Open UI for both clusters in different tabs on your browser,
    3. Press "Deploy changes" in both tabs, but do not confirm yet.
    4. As quick as possible press "Deploy" button in both tabs to start deploy the clusters.

As the result: the first cluster where "Deploy" button was pressed, failed with the notification in UI "Cobbler error".
Provisioning of nodes hasn't even started.

{"build_id": "2014-11-17_13-47-05", "ostf_sha": "82465a94eed4eff1fc8d8e1f2fb7e9993c22f068", "build_number": "502", "auth_required": true, "api": "1.0", "nailgun_sha": "87100dca641d76fd560eac7f7894d9413d8c186d", "production": "docker", "fuelmain_sha": "b585c7082511936ca3ac27e7ed12d1e2386feb90", "astute_sha": "d4fd7048befd22feecf5ba40f17981df5b608621", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-11-17_13-47-05", "ostf_sha": "82465a94eed4eff1fc8d8e1f2fb7e9993c22f068", "build_number": "502", "api": "1.0", "nailgun_sha": "87100dca641d76fd560eac7f7894d9413d8c186d", "production": "docker", "fuelmain_sha": "b585c7082511936ca3ac27e7ed12d1e2386feb90", "astute_sha": "d4fd7048befd22feecf5ba40f17981df5b608621", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "0f51ec2c95ea031ef1190b86d336bee5779b7ed7"}}}, "fuellib_sha": "0f51ec2c95ea031ef1190b86d336bee5779b7ed7"}

Revision history for this message
Dennis Dmitriev (ddmitriev) wrote :
Revision history for this message
Dennis Dmitriev (ddmitriev) wrote :
Revision history for this message
Dennis Dmitriev (ddmitriev) wrote :

Workaround that was proposed in the review https://review.openstack.org/#/c/134929/ didn't help in this case.

Nastya Urlapova (aurlapova)
Changed in fuel:
milestone: 6.1 → 6.0
Aleksey Kasatkin (alekseyk-ru)
Changed in fuel:
assignee: nobody → Fuel Python Team (fuel-python)
Revision history for this message
Dima Shulyak (dshulyak) wrote :

@Dennis, it looks like that this bug is duplicate of https://bugs.launchpad.net/fuel/+bug/1387699

Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Fuel Astute Team (fuel-astute)
importance: Undecided → High
status: New → Confirmed
Revision history for this message
Vladimir Sharshov (vsharshov) wrote :

Looks like reproducing this bug: https://bugs.launchpad.net/fuel/+bug/1318716

Yes, nature of this bug also looks like cobbler overloading (maybe race condition in case of parallel operation).

Changed in fuel:
assignee: Fuel Astute Team (fuel-astute) → Vladimir Sharshov (vsharshov)
OpenStack Infra (hudson-openstack)
Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-astute (master)

Reviewed: https://review.openstack.org/134929
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=0bcc91185984a1f9e4bd68d9f247266c35349dae
Submitter: Jenkins
Branch: master

commit 0bcc91185984a1f9e4bd68d9f247266c35349dae
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Mon Nov 17 16:01:14 2014 +0300

    Add retries in case of Cobbler overloading

    Offen if Cobbler run two big task in parallel
    we got error based on Cobbler overloading.
    If we wait and try request again, it will
    process without problem. This change
    implement such behavior.

    Change-Id: I1e9774b9398dfb502159d7ba21e7e5b55b75b3a1
    Closes-Bug: #1387699
    Closes-Bug: #1396181

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-astute (stable/5.1)

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/137553

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-astute (stable/5.1)

Reviewed: https://review.openstack.org/137553
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=ef8aa0fd0e3ce20709612906f1f0551b5682a6ce
Submitter: Jenkins
Branch: stable/5.1

commit ef8aa0fd0e3ce20709612906f1f0551b5682a6ce
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Mon Nov 17 16:01:14 2014 +0300

    Add retries in case of Cobbler overloading

    Offen if Cobbler run two big task in parallel
    we got error based on Cobbler overloading.
    If we wait and try request again, it will
    process without problem. This change
    implement such behavior.

    Change-Id: I1e9774b9398dfb502159d7ba21e7e5b55b75b3a1
    Closes-Bug: #1387699
    Closes-Bug: #1396181
    (cherry picked from commit 0bcc91185984a1f9e4bd68d9f247266c35349dae)

Revision history for this message
Anastasia Palkina (apalkina) wrote :

Cannot reproduce on latest ISOs for 6.0 #49 and #56

"build_id": "2014-12-18_01-32-01", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "56", "auth_required": true, "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-18_01-32-01", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "56", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"}}}, "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"

Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.