Rabbit app failed to start and join cluster at the second controller node but cannot be noticed by OCF logic

http://jenkins-product.srt.mirantis.net:8080/view/6.1_swarm/job/6.1.system_test.centos.bonding_ha/125/testReport/(root)/deploy_bonding_ha_active_backup/deploy_bonding_ha_active_backup/

But main issue looks like with the rabbit@node-2 refused to start at 2015-05-16T01:57:17 but it looks like OCF logic was not able to detect and handle the failure as appropriate

Fix proposed to branch: master
Review: https://review.openstack.org/184070

Reviewed: https://review.openstack.org/184070
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=454a1485453919756b8a4e3083fb71b0928158ec
Submitter: Jenkins
Branch: master

commit 454a1485453919756b8a4e3083fb71b0928158ec
Author: Bogdan Dobrelya <email address hidden>
Date: Mon May 18 16:00:37 2015 +0200

    Fix get_monitor function and local vars for rabbit OCF

    W/o this fix:
    * get_monitor() does not handle the case then rabbit
    app is running and reports $OCF_RUNNING. This makes
    the resource to stuck in semi-started state for ever.
    * get_monitor() does not always record in logs when
    sets master score 0 to ensure the slave does not get
    promoted. This makes logging patterns not consistent.
    * get_monitor() has two exit point for generic error case
    instead of one. This makes logging patterns not consistent.
    * some variables are missing the local
    declaration and might be reused in another places
    leading to unpredictable results. For example,
    there is the nodelist var should be local as it is
    used both action_notify(), get_monitor().

    The solution is:
    * fix missing local declarations for variables and
    remove unused ones as well.
    * add missing log record for "ensuring this slave does
    not get promoted."
    * handle the get_monitor() case when rabbit app is not
    running as appropriate and report $OCF_NOT_RUNNING.
    * make the single get_monitor() return point for the
    $OCF_ERR_GENERIC case.

    Closes-bug: #1455761

    Change-Id: Iba5e9d984083acea1392cad0abd6453f5d6fbf8b
    Signed-off-by: Bogdan Dobrelya <email address hidden>

How to reproduce:
root@node-8:~# rabbitmqctl stop_app
Stopping node 'rabbit@node-8' ...
...done.
root@node-8:~# ocf_handler_rabbitmq-server monitor
...
Exit status: -e Success (0)

How to test:
With the fix applied, the command above should report Not running instead of Success

Master logic was not fixed (cannot pass the test), reopening

The issue is that by design, the start action will start beam process and stop the rabbit app, unless master promoted or slave joined cluster.
This means that in order to fix this issue, the start action must be redesigned as the following:
1) Do not remove the iptables block rule on the action start() exit.
2) Leave the rabbit app started on the action start() exit.
3) remove the iptables block rule either on the post-promote notify, when master is elected and ready to join other nodes; or on the post-start notify, when slave is ready to join the cluster.

Otherwise, this bug cannot be fixed. We have to revert https://review.openstack.org/184070 as it introduced regression to action monitor

Related fix proposed to branch: master
Review: https://review.openstack.org/184239

Raised to critical due to revert of the regression is required

Reviewed: https://review.openstack.org/184239
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=5cc384073df35300501b161f05cf45f816d7703d
Submitter: Jenkins
Branch: master

commit 5cc384073df35300501b161f05cf45f816d7703d
Author: Bogdan Dobrelya <email address hidden>
Date: Tue May 19 14:51:20 2015 +0200

    Revert "Merge "Fix get_monitor function and local vars for rabbit OCF""

    This reverts commit 0562e2ae27c1d6b2e027e63c5af4e1153f44224b, reversing
    changes made to 5fd379f0cd8106808ab9fc098ca1094c7d91041a.

    The issue is that by design, the start action will start beam process
    and stop the rabbit app, unless master promoted or slave joined cluster.
    We have to revert aforementioned commit as it introduced a regression
    to the action monitor, which is to report "Not running", when rabbit app
    is stopped.

    Related-bug: #1455761
    Change-Id: I686c620092909351c910598aeef0dcc73d2da080

As we discussed with Vladimir Kuklin, the solution should be either to:
* redesign OCF start action to not stop rabbit app on exit (can't be done for the 6.1 as too risky)
* or, to modify post-start notify logic to process join cluster both by nodes being started AND already running ones

Fix proposed to branch: master
Review: https://review.openstack.org/184671

Related fix proposed to branch: master
Review: https://review.openstack.org/184674

note: how to test the patch https://review.openstack.org/184674 at the deployed env -
rabbitmqctl eval 'application:get_env(rabbit, mnesia_table_loading_timeout).'
should return 10000 value

Reviewed: https://review.openstack.org/184674
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=a2a146972dc63208280920ff1cf9321a6300171b
Submitter: Jenkins
Branch: master

commit a2a146972dc63208280920ff1cf9321a6300171b
Author: Vladimir Kuklin <email address hidden>
Date: Thu May 21 05:04:57 2015 +0300

    Set mnesia_table_loading_timeout to 10 seconds

    This commit sets mnesia_table_loading_timeout to
    10 seconds thus making rabbitmq cluster failover
    process faster. This option was initially suggested
    by Michael Klishin (RabbitMQ developer)

    Change-Id: I8ff6388cdd785404ea3659584b20b9e977a1c253
    Related-bug: #1455761
    Related-bug: #1432603

Reviewed: https://review.openstack.org/184671
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=db6ec616bf3b4dea7ee8e7583f12fad16e501d72
Submitter: Jenkins
Branch: master

commit db6ec616bf3b4dea7ee8e7583f12fad16e501d72
Author: Vladimir Kuklin <email address hidden>
Date: Thu May 21 03:50:24 2015 +0300

    Check hostlist against starting and active resources

    This commit makes post-start notify action to check
    hostlist of nodes that should be joined to the cluster
    to contain not only nodes that will be started but
    also ones that are already started. This fixes
    the case when Pacemaker sends notifies only for
    the latest event and thus the node which is not
    included into the start list will not join the
    cluster. Also it checks whether the node is
    already clustered and skips the join if it
    is not needed.

    Change-Id: Ibe8ecdcfe42c14228350b1eb3c9d08b1a64e117d
    Closes-bug: #1455761

Folks, after we merged https://review.openstack.org/184671,
there is a situation possible when the action start joins the node to the cluster (it becomes visible in db_nodes), but the monitor action cannot see if it is absent in running_db_nodes!

root@node-1:~# rabbitmqctl eval 'mnesia:system_info(running_db_nodes).'
['rabbit@node-3','rabbit@node-2']
...done.
root@node-1:~# rabbitmqctl eval 'mnesia:system_info(db_nodes).'
['rabbit@node-2','rabbit@node-3','rabbit@node-1']
...done.

root@node-1:~# ocf_handler_rabbitmq-server monitor
lrmd: DEBUG: p_rabbitmq-server: monitor: action start.
lrmd: INFO: p_rabbitmq-server: get_monitor(): get_status() returns 0.
lrmd: INFO: p_rabbitmq-server: get_monitor(): also checking if we are master.
lrmd: INFO: p_rabbitmq-server: get_monitor(): master attribute is 1
lrmd: INFO: p_rabbitmq-server: get_monitor(): checking if rabbit app is running
lrmd: INFO: p_rabbitmq-server: get_monitor(): preparing to update master score for node
lrmd: INFO: p_rabbitmq-server: get_monitor(): comparing our uptime (0) with node-3.test.domain.local (787)
lrmd: INFO: p_rabbitmq-server: get_monitor(): get_monitor function ready to return 0
lrmd: DEBUG: p_rabbitmq-server: monitor: role:
lrmd: DEBUG: p_rabbitmq-server: monitor: result: 0
lrmd: DEBUG: p_rabbitmq-server: monitor: action end.
Exit status: -e Success (0)

Fix proposed to branch: master
Review: https://review.openstack.org/184987

It looks like that the problem is just in the OCF_RESKEY_CRM_meta_notify_master_uname value can be sometimes returned empty instead of the current master. If so, the node will miss all join events and will remain unjoined.

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: master
Review: https://review.openstack.org/184987

Fix proposed to branch: master
Review: https://review.openstack.org/185044

The impact is not critical as the issue does not introduce full downtime for the AMQP cluster

Reviewed: https://review.openstack.org/179032
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=c93bfdc5a70459432ca1989e3f5dccf44885c75d
Submitter: Jenkins
Branch: master

commit c93bfdc5a70459432ca1989e3f5dccf44885c75d
Author: Bogdan Dobrelya <email address hidden>
Date: Thu Apr 30 13:56:18 2015 +0200

    Disable rabbitmq management plugin

    There is a known security issue exist in the
    management plugin for RabbitMQ <3.4.3, so it
    has to be disabled by default.

    Closes-bug: #1450443
    Closes-bug: #1455761

    Change-Id: Ic01c26200f6019a8112b1c5fb04a282e64b3b3e6
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Reviewed: https://review.openstack.org/185044
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=085fe8c5a2255d4274cdcee5c2a74c15c443c0db
Submitter: Jenkins
Branch: master

commit 085fe8c5a2255d4274cdcee5c2a74c15c443c0db
Author: Bogdan Dobrelya <email address hidden>
Date: Fri May 22 16:40:16 2015 +0200

    Fix rabbit OCF demote/stop/promote actions

    * When the rabbit node went down, its status remains 'running'
      in mnesia db for a while, so few retries (50 sec of total) are
      required in order to kick and forget this node from the cluster.
      This also requires +50 sec for actions stop & demote timeout.
    * The rabbit master score in the CIB is retained after the current
      master moved manually. This is wrong and the score must be reset
      ASAP for post-demote and post-stop as well.
    * The demoted node must be kicked from cluster by other nodes
      on post-demote processing.
    * Post-demote should stop the rabbit app at the node being demoted as
      this node should be kicked from the cluster by other nodes.
      Instead, it stops the app at the *other* nodes and brings full
      cluster downtime.
    * The check to join should be only done at the post-start and not at
      the post-promote, otherwise the node being promoted may think it
      is clustered with some node while the join check reports it as
      already clustered with another one.
      (the regression was caused by https://review.openstack.org/184671)
    * Change `hostname` call to `crm_node -n` via $THIS_PCMK_NODE
      everywhere to ensure we are using correct pacemaker node name
    * Handle empty values for OCF_RESKEY_CRM_meta_notify_* by reporting
      the resource as not running. This will rerun resource and restore
      its state, eventually.

    Closes-bug: #1436812
    Closes-bug: #1455761

    Change-Id: Ib01c1731b4f06e6b643a4bca845828f7db507ad3
    Signed-off-by: Bogdan Dobrelya <email address hidden>

The original bug is resolved now. The remaining issue with missing start for the rabbit app should be addressed by https://review.openstack.org/185530 as a separate bug

The new bug is https://bugs.launchpad.net/fuel/+bug/1458828

Setting this as Won't Fix for 5.1.1-updates and 6.0-updates, as such a complex change cannot be delivered in the scope of the Maintenance Update. Also, the possible solution of the backporting of RabbitMQ OCF script is covered in details by the Operations Guide from the official documentation of the Product.