Comment 1 for bug 2000107

Revision history for this message
Rafael Lopez (rafael.lopez) wrote (last edit ):

This is reproducible by triggering an SST to a secondary node. One way to do this:

1. Deploy charm using 3 units with min-size 2, eg.
juju deploy -n 3 --series bionic percona-cluster --config min-cluster-size=2

2. Log into a non-leader node, stop mysql, delete /var/lib/percona-xtradb-cluster/grastate.dat, and restart
juju ssh {non leader}
sudo systemctl stop mysql
sudo rm /var/lib/percona-xtradb-cluster/grastate.dat
sudo systemctl start mysql

This will trigger an SST which juju does not know about, which wipes out most of /var/lib/percona-xtradb-cluster/ (normal operation, as documented in [1]), including the 'seeded' file.

3. After the database completes the SST the mysql percona cluster is completely recovered. Checking /var/lib/percona-xtradb-cluster/seeded on the non-leader node, it is missing and juju status will show the unit stuck waiting to bootstrap as follows:
Unit Workload Agent Machine Public address Ports Message
percona-cluster/0* active idle 37 10.133.201.64 3306/tcp Unit is ready
percona-cluster/1 waiting idle 38 10.133.201.185 3306/tcp Unit waiting to bootstrap ('seeded' file missing)
percona-cluster/2 active idle 39 10.133.201.124 3306/tcp Unit is ready

[1] https://docs.percona.com/percona-xtradb-cluster/5.7/manual/xtrabackup_sst.html