Initial log parsing

The cluster comes up fine before vault sends out certificates. Once the cluster receives certificates the cluster goes into a rolling restart as expected.

However, the first node to restart reports problems immediately with "Failed to stop the group replication applier thread" during shutdown

Here is a sorted (interleaved grep from the mysql error logs:
Formatted version: https://pastebin.ubuntu.com/p/xf572df7vD/

mysql-innodb-cluster_0/var/log/mysql/error.log 2021-04-28T11:20:44.880786Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.23-0ubuntu0.20.04.1).:
mysql-innodb-cluster_0/var/log/mysql/error.log 2021-04-28T11:20:45.073488Z 11 [ERROR] [MY-011535] [Repl] Plugin group_replication reported: 'Failed to stop the group replication applier thread.':
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:20:48.510785Z 0 [Warning] [MY-011499] [Repl] Plugin group_replication reported: 'Members removed from the group: 192.168.33.54
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:20:48.515514Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 192.168.33.52:3306, 192.168.33.55
mysql-innodb-cluster_2/var/log/mysql/error.log 2021-04-28T11:20:48.526308Z 0 [Warning] [MY-011499] [Repl] Plugin group_replication reported: 'Members removed from the group: 192.168.33.54
mysql-innodb-cluster_2/var/log/mysql/error.log 2021-04-28T11:20:48.527036Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 192.168.33.52:3306, 192.168.33.55
mysql-innodb-cluster_0/var/log/mysql/error.log 2021-04-28T11:20:50.688199Z 0 [System] [MY-011504] [Repl] Plugin group_replication reported: 'Group membership changed: This member has left the group.'
mysql-innodb-cluster_0/var/log/mysql/error.log 2021-04-28T11:20:50.695954Z 0 [System] [MY-011651] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' has been stopped.':
mysql-innodb-cluster_0/var/log/mysql/error.log 2021-04-28T11:20:56.752996Z 0 [System] [MY-013587] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' is starting.':
mysql-innodb-cluster_0/var/log/mysql/error.log 2021-04-28T11:20:58.223924Z 10 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_applier' executed'. Previous state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.::
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:21:00.623164Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 192.168.33.52:3306, 192.168.33.55
mysql-innodb-cluster_0/var/log/mysql/error.log 2021-04-28T11:21:00.626408Z 2 [System] [MY-011511] [Repl] Plugin group_replication reported: 'This server is working as secondary member with primary member address 192.168.33.52:3306.'
mysql-innodb-cluster_2/var/log/mysql/error.log 2021-04-28T11:21:00.630934Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 192.168.33.52:3306, 192.168.33.55
mysql-innodb-cluster_0/var/log/mysql/error.log 2021-04-28T11:21:01.649666Z 0 [System] [MY-013471] [Repl] Plugin group_replication reported: 'Distributed recovery will transfer data using: Incremental recovery from a group donor'
mysql-innodb-cluster_0/var/log/mysql/error.log 2021-04-28T11:21:01.655107Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 192.168.33.52:3306, 192.168.33.55
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:22:18.998153Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.23-0ubuntu0.20.04.1).:
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:22:19.114028Z 11 [ERROR] [MY-011535] [Repl] Plugin group_replication reported: 'Failed to stop the group replication applier thread.':
mysql-innodb-cluster_0/var/log/mysql/error.log 2021-04-28T11:22:22.124782Z 0 [Warning] [MY-011499] [Repl] Plugin group_replication reported: 'Members removed from the group: 192.168.33.55
mysql-innodb-cluster_0/var/log/mysql/error.log 2021-04-28T11:22:22.124896Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 192.168.33.52:3306, 192.168.33.54
mysql-innodb-cluster_2/var/log/mysql/error.log 2021-04-28T11:22:22.125859Z 0 [Warning] [MY-011499] [Repl] Plugin group_replication reported: 'Members removed from the group: 192.168.33.55
mysql-innodb-cluster_2/var/log/mysql/error.log 2021-04-28T11:22:22.125958Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 192.168.33.52:3306, 192.168.33.54
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:22:24.242854Z 0 [System] [MY-011504] [Repl] Plugin group_replication reported: 'Group membership changed: This member has left the group.'
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:22:24.246264Z 0 [System] [MY-011651] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' has been stopped.':
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:22:28.327116Z 0 [System] [MY-013587] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' is starting.':
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:22:28.968854Z 10 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_applier' executed'. Previous state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.::
mysql-innodb-cluster_0/var/log/mysql/error.log 2021-04-28T11:22:30.946956Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 192.168.33.52:3306, 192.168.33.55
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:22:30.948047Z 2 [System] [MY-011511] [Repl] Plugin group_replication reported: 'This server is working as secondary member with primary member address 192.168.33.52:3306.'
mysql-innodb-cluster_2/var/log/mysql/error.log 2021-04-28T11:22:30.950223Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 192.168.33.52:3306, 192.168.33.55
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:22:31.950612Z 0 [System] [MY-013471] [Repl] Plugin group_replication reported: 'Distributed recovery will transfer data using: Incremental recovery from a group donor'
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:22:31.952268Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 192.168.33.52:3306, 192.168.33.55
mysql-innodb-cluster_2/var/log/mysql/error.log 2021-04-28T11:22:49.088142Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.23-0ubuntu0.20.04.1).:
mysql-innodb-cluster_0/var/log/mysql/error.log 2021-04-28T11:22:52.641130Z 0 [Warning] [MY-011499] [Repl] Plugin group_replication reported: 'Members removed from the group: 192.168.33.52
mysql-innodb-cluster_0/var/log/mysql/error.log 2021-04-28T11:22:52.641166Z 0 [System] [MY-011500] [Repl] Plugin group_replication reported: 'Primary server with address 192.168.33.52:3306 left the group. Electing new Primary.'
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:22:52.641220Z 0 [Warning] [MY-011499] [Repl] Plugin group_replication reported: 'Members removed from the group: 192.168.33.52
mysql-innodb-cluster_0/var/log/mysql/error.log 2021-04-28T11:22:52.641234Z 0 [Warning] [MY-011512] [Repl] Plugin group_replication reported: 'Unable to set any member as primary. No suitable candidate.':
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:22:52.641267Z 0 [System] [MY-011500] [Repl] Plugin group_replication reported: 'Primary server with address 192.168.33.52:3306 left the group. Electing new Primary.'
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:22:52.641417Z 0 [Warning] [MY-011512] [Repl] Plugin group_replication reported: 'Unable to set any member as primary. No suitable candidate.':
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:22:53.642661Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 192.168.33.55:3306, 192.168.33.54
mysql-innodb-cluster_0/var/log/mysql/error.log 2021-04-28T11:22:53.643062Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 192.168.33.55:3306, 192.168.33.54
mysql-innodb-cluster_2/var/log/mysql/error.log 2021-04-28T11:22:54.768072Z 0 [System] [MY-011504] [Repl] Plugin group_replication reported: 'Group membership changed: This member has left the group.'
mysql-innodb-cluster_2/var/log/mysql/error.log 2021-04-28T11:22:54.771359Z 0 [System] [MY-011651] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' has been stopped.':
mysql-innodb-cluster_2/var/log/mysql/error.log 2021-04-28T11:22:59.568274Z 0 [System] [MY-013587] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' is starting.':
mysql-innodb-cluster_2/var/log/mysql/error.log 2021-04-28T11:23:00.368795Z 19 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_applier' executed'. Previous state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.::
mysql-innodb-cluster_2/var/log/mysql/error.log 2021-04-28T11:23:02.268857Z 0 [ERROR] [MY-011526] [Repl] Plugin group_replication reported: 'This member has more executed transactions than those present in the group. Local transactions: 5ddbba38-a808-11eb-91b0-00163e82404d
mysql-innodb-cluster_2/var/log/mysql/error.log 2021-04-28T11:23:02.268984Z 0 [ERROR] [MY-011522] [Repl] Plugin group_replication reported: 'The member contains transactions not present in the group. The member will now exit the group.':
mysql-innodb-cluster_2/var/log/mysql/error.log 2021-04-28T11:23:02.269024Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 192.168.33.52:3306, 192.168.33.55
mysql-innodb-cluster_0/var/log/mysql/error.log 2021-04-28T11:23:02.269556Z 0 [Warning] [MY-011512] [Repl] Plugin group_replication reported: 'Unable to set any member as primary. No suitable candidate.':
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:23:02.271344Z 0 [Warning] [MY-011512] [Repl] Plugin group_replication reported: 'Unable to set any member as primary. No suitable candidate.':
mysql-innodb-cluster_0/var/log/mysql/error.log 2021-04-28T11:23:03.275743Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 192.168.33.52:3306, 192.168.33.55
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:23:03.280878Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 192.168.33.52:3306, 192.168.33.55
mysql-innodb-cluster_0/var/log/mysql/error.log 2021-04-28T11:23:04.291181Z 0 [Warning] [MY-011499] [Repl] Plugin group_replication reported: 'Members removed from the group: 192.168.33.52
mysql-innodb-cluster_0/var/log/mysql/error.log 2021-04-28T11:23:04.291281Z 0 [Warning] [MY-011512] [Repl] Plugin group_replication reported: 'Unable to set any member as primary. No suitable candidate.':
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:23:04.293165Z 0 [Warning] [MY-011499] [Repl] Plugin group_replication reported: 'Members removed from the group: 192.168.33.52
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:23:04.293364Z 0 [Warning] [MY-011512] [Repl] Plugin group_replication reported: 'Unable to set any member as primary. No suitable candidate.':
mysql-innodb-cluster_0/var/log/mysql/error.log 2021-04-28T11:23:05.294435Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 192.168.33.55:3306, 192.168.33.54
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:23:05.294855Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 192.168.33.55:3306, 192.168.33.54
mysql-innodb-cluster_2/var/log/mysql/error.log 2021-04-28T11:23:06.224372Z 0 [System] [MY-011504] [Repl] Plugin group_replication reported: 'Group membership changed: This member has left the group.'
mysql-innodb-cluster_0/var/log/mysql/error.log 2021-04-28T11:33:42.076811Z 21 [ERROR] [MY-011574] [Repl] Plugin group_replication reported: 'Maximum number of retries when trying to connect to a donor reached. Aborting group replication incremental recovery.':
mysql-innodb-cluster_0/var/log/mysql/error.log 2021-04-28T11:33:42.076917Z 21 [ERROR] [MY-011620] [Repl] Plugin group_replication reported: 'Fatal error during the incremental recovery process of Group Replication. The server will leave the group.':
mysql-innodb-cluster_0/var/log/mysql/error.log 2021-04-28T11:33:42.077014Z 21 [ERROR] [MY-011712] [Repl] Plugin group_replication reported: 'The server was automatically set into read only mode after an error was detected.':
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:33:43.994039Z 0 [Warning] [MY-011499] [Repl] Plugin group_replication reported: 'Members removed from the group: 192.168.33.54
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:33:44.995974Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 192.168.33.55:3306 on view 16196040323665586
mysql-innodb-cluster_0/var/log/mysql/error.log 2021-04-28T11:33:45.459239Z 0 [System] [MY-011504] [Repl] Plugin group_replication reported: 'Group membership changed: This member has left the group.'
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:34:04.977160Z 22 [ERROR] [MY-011575] [Repl] Plugin group_replication reported: 'All donors left. Aborting group replication incremental recovery.':
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:34:04.977321Z 22 [ERROR] [MY-011620] [Repl] Plugin group_replication reported: 'Fatal error during the incremental recovery process of Group Replication. The server will leave the group.':
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:34:04.978371Z 22 [ERROR] [MY-011712] [Repl] Plugin group_replication reported: 'The server was automatically set into read only mode after an error was detected.':
mysql-innodb-cluster_1/var/log/mysql/error.log 2021-04-28T11:34:08.296074Z 0 [System] [MY-011504] [Repl] Plugin group_replication reported: 'Group membership changed: This member has left the group.'

Things I noticed:
The leader node went last (but that *should* not matter

This could be an asynchronous process in mysql that is racing the charm.

I'll keep digging.