Rabbit join race with OSTF tests 'RabbitMQ availability' and 'RabbitMQ replication' are failed after reschedule router from primary controller and destroying it

How long did you wait for rabbit failover to complete after the step 8?

Bogdan, we wait a new agent
https://github.com/stackforge/fuel-qa/blob/ac40a9f6d92e0f1d54371b9f3c2a6d4dc018eff2/fuelweb_test/tests/tests_strength/test_neutron.py#L340

the test is incorrect. After you destroyed any controller, you should wait for rabbit failover to finish, which is indicated by passed OSTF HA tests

@bogdando, we wait until OSTF HA tests pass [1] it happens between 8 and 9 steps

[1] https://github.com/stackforge/fuel-qa/blob/ac40a9f6d92e0f1d54371b9f3c2a6d4dc018eff2/fuelweb_test/tests/tests_strength/test_neutron.py#L333

can you clarify if this is temporal service interruption or permanent outage?

@maximov, we destroy node - https://github.com/stackforge/fuel-qa/blob/71abe120b45480ac04ec0f4adf685f9787bfc139/fuelweb_test/tests/tests_strength/test_neutron.py#L327-L328

@Andrew, the failover had stuck and downtime was a permanent.
The issue is that a last-man-standing node should not be restarted by the "there are no nodes to join to" logic. Otherwise there is a race like this one might happen. Snippet http://pastebin.com/N36fX8CX

Raised to critical as this race condition is a major defect in the OCF agent logic

Fix proposed to branch: master
Review: https://review.openstack.org/220103

Reviewed: https://review.openstack.org/220103
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=403b28c2aff8aa9f37125d3ff1ac09861990da7e
Submitter: Jenkins
Branch: master

commit 403b28c2aff8aa9f37125d3ff1ac09861990da7e
Author: Bogdan Dobrelya <email address hidden>
Date: Thu Sep 3 13:53:13 2015 +0200

    Detect a last man standing for rabbit OCF agent

    W/o this patch, the race condition is possible
    when there is no running rabbit nodes/resource
    master. The rabbit nodes will start/stop in an
    endless loop as a result introducing full downtime
    for AMQP cluster and cloud control plane.

    The solution is:
    * On post-start/post-promote notify, do nothing, if
      either of the following is a true:
      - there is no rabbit resources running or no master
      - the list of rabbit resources being started/promoted
        reported empty
    * For such cases, do not report resource failure and delegate
      recovery, if needed, to the "running out of the cluster"
      monitor's logic.
    * Additionally, report about a last man standing when
      there is no running rabbit resources around.

    Closes-bug: #1491306

    Change-Id: If1c62fac26b63410636413c49fce55c35e53dc5f
    Signed-off-by: Bogdan Dobrelya <email address hidden>

verified for 7.0
{"build_id": "288", "build_number": "288", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "288", "build_number": "288", "api": "1.0", "fuel-library_sha": "121016a09b0e889994118aa3ea42fa67eabb8f25", "nailgun_sha": "93477f9b42c5a5e0506248659f40bebc9ac23943", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d", "production": "docker", "python-fuelclient_sha": "1ce8ecd8beb640f2f62f73435f4e18d1469979ac", "astute_sha": "a717657232721a7fafc67ff5e1c696c9dbeb0b95", "fuel-ostf_sha": "1f08e6e71021179b9881a824d9c999957fcc7045", "release": "7.0", "fuelmain_sha": "6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "121016a09b0e889994118aa3ea42fa67eabb8f25", "nailgun_sha": "93477f9b42c5a5e0506248659f40bebc9ac23943", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d", "production": "docker", "python-fuelclient_sha": "1ce8ecd8beb640f2f62f73435f4e18d1469979ac", "astute_sha": "a717657232721a7fafc67ff5e1c696c9dbeb0b95", "fuel-ostf_sha": "1f08e6e71021179b9881a824d9c999957fcc7045", "release": "7.0", "fuelmain_sha": "6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85"}

Setting this as Won't Fix for 5.1.1-updates and 6.0-updates, as such a complex change cannot be delivered in the scope of the Maintenance Update. Also, the possible solution of the backporting of RabbitMQ OCF script is covered in details by the Operations Guide from the official documentation of the Product.

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/239448

Reviewed: https://review.openstack.org/239448
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=e052a2d5cf2d313853b472b971033f1c83c4d55c
Submitter: Jenkins
Branch: stable/6.1

commit e052a2d5cf2d313853b472b971033f1c83c4d55c
Author: Bogdan Dobrelya <email address hidden>
Date: Thu Sep 3 13:53:13 2015 +0200

    Detect a last man standing for rabbit OCF agent

    W/o this patch, the race condition is possible
    when there is no running rabbit nodes/resource
    master. The rabbit nodes will start/stop in an
    endless loop as a result introducing full downtime
    for AMQP cluster and cloud control plane.

    The solution is:
    * On post-start/post-promote notify, do nothing, if
      either of the following is a true:
      - there is no rabbit resources running or no master
      - the list of rabbit resources being started/promoted
        reported empty
    * For such cases, do not report resource failure and delegate
      recovery, if needed, to the "running out of the cluster"
      monitor's logic.
    * Additionally, report about a last man standing when
      there is no running rabbit resources around.

    Closes-bug: #1491306

    Conflicts:
     files/fuel-ha-utils/ocf/rabbitmq

    Change-Id: If1c62fac26b63410636413c49fce55c35e53dc5f