Fuel for OpenStack

Deploy of detached rabbit plugin failed with error: Loop detected!

Bug #1651066 reported by Nastya Urlapova on 2016-12-19

This bug affects 2 people

	Status	Importance	Assigned to	Milestone
Fuel for OpenStack	Fix Committed	High	Dmitry Mescheryakov	Fuel for OpenStack 11.0
Mitaka	Fix Released	High	Dmitry Mescheryakov	Fuel for OpenStack 9.2
Newton	Fix Committed	High	Dmitry Ilyin	Fuel for OpenStack 10.x-updates

Bug Description

Deployment with separate rabbitmq nodes and ceph for all

    Scenario:
        1. Install the plugin on the master node
        2. Create Ubuntu, Neutron VXLAN, ceph for all storages cluster
        3. Change ceph replication factor to 2
        4. Add 3 nodes with controller role
        5. Add 3 nodes with rabbitmq role
        6. Add 1 compute node
        7. Add 2 ceph node
        8. Run network verification
        >>> 9. Deploy changes <<< failed here
        10. Run network verification
        11. Run OSTF tests

Deployment failed with error:
Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. Method task_deploy. Cluster[]: Loop detected! Path: Task[generate_keys/master], Task[copy_keys/1], Task[pre_deployment_end/virtual_sync_node], Task[deploy_start/3], Task[hiera/3], Task[globals/3], Task[rabbitmq-hiera-override/3], Task[logging/3], Task[tools/3], Task[netconfig/3], Task[firewall/3], Task[primary-cluster/3], Task[cluster/1], Task[virtual_ips/1], Task[netconfig/3].

Version9.2 snapshot #638

Tags:

Revision history for this message

Nastya Urlapova (aurlapova) wrote on 2016-12-19:

fail_error_separate_rabbit_ceph_service-fuel-snapshot-2016-12-16_10-47-19.tar Edit (13.6 MiB, application/x-tar)

Changed in fuel:
milestone:	none → 9.2
assignee:	nobody → Fuel Sustaining (fuel-sustaining-team)
tags:	added: swarm-fail

Dmitry Mescheryakov (dmitrymex) on 2016-12-19

Changed in fuel:
assignee:	Fuel Sustaining (fuel-sustaining-team) → Dmitry Mescheryakov (dmitrymex)

Vitaly Sedelnik (vsedelnik) on 2016-12-19

Changed in fuel:
status:	New → Confirmed
tags:	added: area-plugins

Revision history for this message

Dmitry Belyaninov (dbelyaninov) wrote on 2016-12-20:

failed job
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.plugins.detach-rabbitmq-ci-group/110/testReport/(root)/separate_rabbit_service/separate_rabbit_service/

similar bug
https://bugs.launchpad.net/fuel/+bug/1649889

Tatyana Borukhovich (tatyanaborukhovich) on 2016-12-27

tags:

added: swarm-blocker

Tatyana Borukhovich (tatyanaborukhovich) on 2016-12-27

tags:

added: blocker-for-qa

Revision history for this message

Dmitry Belyaninov (dbelyaninov) wrote on 2016-12-29:

https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.plugins.thread_2_separate_services/161/testReport/(root)/separate_rabbit_service/separate_rabbit_service/

Revision history for this message

Dmitry Belyaninov (dbelyaninov) wrote on 2016-12-30:

https://product-ci.infra.mirantis.net/job/9.x.acceptance.ubuntu.plugins.thread_separate_services_ceph/30/testReport/(root)/separate_db_ceph_service/separate_db_ceph_service/

Tatyana Borukhovich (tatyanaborukhovich) on 2017-01-04

tags:

removed: swarm-blocker

Revision history for this message

Nastya Urlapova (aurlapova) wrote on 2017-01-09:

So issue affects 5 test cases, so it is a swarm-blocker.

tags:

added: swarm-blocker
removed: swarm-fail

Dmitry Mescheryakov (dmitrymex) on 2017-01-11

Changed in fuel:
milestone:	9.2 → 11.0

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-11: Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/418852

Changed in fuel:
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-12: Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/419543

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-15: Fix proposed to fuel-library (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/420434

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-16: Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/418852
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=ceb77edfe06acc3ce703ff704054074aab4a00e4
Submitter: Jenkins
Branch: master

commit ceb77edfe06acc3ce703ff704054074aab4a00e4
Author: Dmitry Mescheryakov <email address hidden>
Date: Wed Jan 11 13:50:30 2017 +0400

Limit cluster task cross-dependency on primary-cluster

    Cluster task should cross-depend on primary-cluster only if current
    node has primary-something role. Otherwise a loop listed in the
    referenced bug occurs and deployment breaks.

    Also, that CR makes change in rabbitmq networking trigger OpenStack
    tasks. That fixes the old assumption that RabbitMQ always lives on
    controllers. Now changes in networking of nodes with RabbitMQ role
    will also trigger modification of configs for OpenStack services on
    other nodes, updating rabbit_hosts parameter.

For the same reason (changes in rabbitmq should trigger OpenStack
tasks) all OpenStack tasks now cross-depend on rabbitmq.

Finally the CR fixes typo in openstack-cinder/tasks.yaml:
cross-depened-by -> cross-dependend-by

Closes-Bug: #1651066

Change-Id: I8b262ef73c9c62ce5a71f0a5eee001b61ba9ecb1

Changed in fuel:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-16: Fix merged to fuel-library (stable/mitaka)

#10

Reviewed: https://review.openstack.org/419543
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=ab1c4dd8260d3d66d6d9a6f13a8a7ea3f9acd934
Submitter: Jenkins
Branch: stable/mitaka

commit ab1c4dd8260d3d66d6d9a6f13a8a7ea3f9acd934
Author: Dmitry Mescheryakov <email address hidden>
Date: Wed Jan 11 13:50:30 2017 +0400

Limit cluster task cross-dependency on primary-cluster

    Cluster task should cross-depend on primary-cluster only if current
    node has primary-something role. Otherwise a loop listed in the
    referenced bug occurs and deployment breaks.

For the same reason (changes in rabbitmq should trigger OpenStack
tasks) all OpenStack tasks now cross-depend on rabbitmq.

Finally the CR fixes typo in openstack-cinder/tasks.yaml:
cross-depened-by -> cross-dependend-by

Closes-Bug: #1651066

Change-Id: I8b262ef73c9c62ce5a71f0a5eee001b61ba9ecb1

Ekaterina Shutova (eshutova) on 2017-01-16

tags:

added: on-verification

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-20: Fix merged to fuel-library (stable/newton)

#11

Reviewed: https://review.openstack.org/420434
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=036fc274b29bdd6e03bf52f5a7e5b4513476ee6d
Submitter: Jenkins
Branch: stable/newton

commit 036fc274b29bdd6e03bf52f5a7e5b4513476ee6d
Author: Dmitry Mescheryakov <email address hidden>
Date: Wed Jan 11 13:50:30 2017 +0400

Limit cluster task cross-dependency on primary-cluster

    Cluster task should cross-depend on primary-cluster only if current
    node has primary-something role. Otherwise a loop listed in the
    referenced bug occurs and deployment breaks.

For the same reason (changes in rabbitmq should trigger OpenStack
tasks) all OpenStack tasks now cross-depend on rabbitmq.

Finally the CR fixes typo in openstack-cinder/tasks.yaml:
cross-depened-by -> cross-dependend-by

Closes-Bug: #1651066

Change-Id: I8b262ef73c9c62ce5a71f0a5eee001b61ba9ecb1

Revision history for this message

Ekaterina Shutova (eshutova) wrote on 2017-01-23:

#12

Verified on 9.2 snapshot #788. Passed run: https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.plugins.detach-rabbitmq-ci-group/135/testReport/
Test is sometimes failed on step: run OSTF tests, but it's another issue.