adding a 3rd hacluster unit frequently makes ha-relation-changed to loop on crm node list

FYI this looks like a race, as we have the same (repeated) deployment
sometimes failing on different hacluster subordinates (keystone,
cinder, glance).

I've seen this several times during deploys. However, the loop on 'crm node list' is not a definitive indicator - there are some cases where pacemaker fails to start on a node which has a correct corosync.conf, and all that is needed is to restart pacemaker.

FYI this has corosync_transport: unicast.

Changed our deployment sequence to deploy the 3 HA
units at once, then relate openstack services, and got
a different issue, on some of them (affected units
changed each time I redeployed , tried couple times):

$ juju run --timeout=10s --service=keystone 'sudo crm status 2>/dev/null|egrep Started:'
- Error: command timed out
  MachineId: 0/lxc/6
  Stdout: ""
  UnitId: keystone/0
- MachineId: 1/lxc/3
  Stdout: ' Started: [ juju-machine-1-lxc-3 juju-machine-2-lxc-3 ]

'
  UnitId: keystone/1
- MachineId: 2/lxc/3
  Stdout: ' Started: [ juju-machine-1-lxc-3 juju-machine-2-lxc-3 ]

'
  UnitId: keystone/2

Logging into the timed out unit shows pacemaker not started,
then hanode-relation-changed looping endlessly on failing
crm node list, after starting pacemaker there the hook could
complete ok: http://paste.ubuntu.com/10454923/

JuanJo or Paul, Do you have any juju logs or syslogs you could attach for analysis? sosreport works as well.

JuanJo have you set the cluster_count config-option in the hacluster charm? If not, when you deploy a 3 node cluster at once, try setting the cluster_count to 3 as well so that the charm is looking to see 3 nodes before it starts the clustering process.

JuanJo

I think I've tracked this problem down to the following situation - I can repro on a 3 node cluster expansion:

1) charms deployed with hacluster with unicast/cluster_count=3 (which is right for an initial three node cluster)

bootstraps fine - cluster forms OK

2) juju add-unit <service>

Additional unit spins - corosync is unable to startup correctly, to crm node list just sits in the loop.

Looking at the debug output of corosync and the local corosync.conf, the new unit only has a partial node list of three nodes, one being itself, and this is confusing the votequorum function.

I tried with what appears to be success to override the expected_votes calculation based on the nodelist with an explicit configuration with votequourm (which was already done for multicast) and I can now expand the cluster reliably.

Hmm - I think that this actually breaks depending on which unit is the current DC owner - the existing units have the new unit in their nodelist, so start to send data, and if the current owner is not in the list, then we get this situation.

Based on that, I still get breaks on new units joining the cluster.

workaround for now is to increase the cluster_count configuration inline with the target cluster size prior to adding units.