Openstack HA , rabbitmq monitoring script should also check for available nodes and clustered nodes

Openstack HA .

Issue found on 2.0-2464 havana build.

Issue was observed when stopping and starting of keepalived on all three controller nodes.

rabbitmq monitoring script should also check for nodes which are up and to that of clustered nodes.

rabbit monitor script shows cluster state and channel state as fine but actually clustering happened only with two nodes.

root@b4s342:/var/log/contrail/ha# ls
cmon-monitor.log rmq-monitor.log token-cleanup.log
root@b4s342:/var/log/contrail/ha# tail -f rmq-monitor.log
12:11:36: INFO: pidof pending rmq channel 30021 and cluster check 30022
12:11:46: INFO: cluster state y and channel state y
12:12:41: INFO: pidof pending rmq channel 9422 and cluster check 9423
12:12:51: INFO: cluster state y and channel state y
12:13:47: INFO: pidof pending rmq channel 21030 and cluster check 21031
12:13:57: INFO: cluster state y and channel state y
12:14:52: INFO: pidof pending rmq channel 371 and cluster check 372
12:15:02: INFO: cluster state y and channel state y
12:15:58: INFO: pidof pending rmq channel 10798 and cluster check 10799
12:16:08: INFO: cluster state y and channel state y

root@b4s342:/var/log/contrail/ha# rabbitmqctl cluster_status
Cluster status of node 'rabbit@b4s342-ctrl' ...
[{nodes,[{disc,['rabbit@b4s342-ctrl','rabbit@b4s343-ctrl',
                'rabbit@b4s344-ctrl']}]},
 {running_nodes,['rabbit@b4s343-ctrl','rabbit@b4s342-ctrl']},
 {cluster_name,<<"rabbit@b4s342">>},
 {partitions,[]}]
...done.
>>>>>>
b4s344 is up but not part of rmq cluster.

root@b4s342:/var/log/contrail/ha#
ot@b4s342:/var/log/contrail/ha# ping b4s344
PING b4s344 (10.84.24.34) 56(84) bytes of data.
64 bytes from b4s344 (10.84.24.34): icmp_req=1 ttl=64 time=0.244 ms
64 bytes from b4s344 (10.84.24.34): icmp_req=2 ttl=64 time=0.110 ms
64 bytes from b4s344 (10.84.24.34): icmp_req=3 ttl=64 time=0.130 ms
^C
--- b4s344 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.110/0.161/0.244/0.059 ms
root@b4s342:/var/log/contrail/ha#