Comment 3 for bug 1732257

What I think happens is that there is a percona-level contention on who's cluster UUID is the right one as we technically ran bootstrap-pxc two times due to a leadership flip to a different unit.

I don't think concurrent bootstrap-pxc is generally a good idea (https://www.percona.com/doc/percona-xtradb-cluster/LATEST/bootstrap.html this doesn't discuss consequential bootstrap-pxc executions on different nodes - I think there is a high chance that the cluster will go crazy like our cluster did).

1.
mysql/2 (the first leader) has set a cluster_state_uuid to b3e1b7e8-c95f-11e7-91fa-56c8581eb616

2017-11-14 17:17:43 DEBUG juju-log Bootstrap PXC Succeeded

2.

mysql/1 - the new leader doesn't error out so it must have generated 0dde54c4-c961-11e7-9775-2f72db67cbda and set is to leader settings.

2017-11-14 17:27:24 DEBUG juju-log Bootstrap PXC Succeeded

3.

mysql/0 - a minion "for life"

Used the correct UUID of the new leader (mysql/1) (0dde54c4-c961-11e7-9775-2f72db67cbda) at first and even started notifying peer units, so:

somehow got past

    if wsrep_ready.lower() in ['on', 'ready']:
        cluster_state_uuid = get_wsrep_value('wsrep_cluster_state_uuid')

and

    elif lead_cluster_state_uuid != cluster_state_uuid:

        log('bootstrap uuid differs: %s != %s' % (lead_cluster_state_uuid,
                                                  cluster_state_uuid),

and got to

    for rid in relation_ids('cluster'):
        notify_bootstrapped(cluster_rid=rid,
                            cluster_uuid=cluster_state_uuid)

2017-11-14 17:31:07 DEBUG juju-log cluster:28: Notifying peers that percona is bootstrapped (uuid=0dde54c4-c961-11e7-9775-2f72db67cbda)

Notice the time stamp: 17:31:07 - this happened way after mysql/2 set up wsrep_cluster_state to b3e1b7e8-c95f-11e7-91fa-56c8581eb616

4. mysql/0 (minion) started saying that wsrep_cluster_uuid == b3e1b7e8-c95f-11e7-91fa-56c8581eb616

2017-11-14 17:31:57 ERROR juju-log bootstrap uuid differs: 0dde54c4-c961-11e7-9775-2f72db67cbda != b3e1b7e8-c95f-11e7-91fa-56c8581eb616