Comment 0 for bug 1942289

Description
==========

Keystone HA is enabled.
After I scale up 1 keystone unit and remove the keystone unit with vip.
The cluster become "Database not initialised".

Versions
===============================
Cloud focal-wallaby

Deploy openstack from https://github.com/openstack-charmers/openstack-bundles
(git commit: 3f74a67704ed7654e0f574ccfd724dc3a224c9f8)

Modify the charm version and turn on ovn in bundle.yaml

diff --git a/stable/openstack-base/bundle.yaml b/stable/openstack-base/bundle.yaml
index 39297cf..c52ecb2 100644
--- a/stable/openstack-base/bundle.yaml
+++ b/stable/openstack-base/bundle.yaml
@@ -219,7 +219,7 @@ applications:
     annotations:
       gui-x: '300'
       gui-y: '1270'
- charm: cs:keystone-323
+ charm: cs:keystone-326
     num_units: 1
     options:
       worker-multiplier: *worker-multiplier
@@ -359,7 +359,7 @@ applications:
     # top of this file.
     options:
       ovn-bridge-mappings: physnet1:br-ex
- bridge-interface-mappings: *data-port
+ #bridge-interface-mappings: *data-port
   vault-mysql-router:
     annotations:
       gui-x: '1535'

Reproduce Step
===================

1. Prepare the Openstack environment like above in serverstack.

2. Turn on Keystone HA

  juju add-unit -n 2 keystone
  juju config keystone vip=10.5.55.1
  juju deploy --config cluster_count=3 hacluster keystone-hacluster
  juju add-relation keystone-hacluster:ha keystone:ha

Everything is still good except the certificate issue in LP: #1930763

3. Scale up 1 keystone unit

  juju add-unit -n 1 keystone

4. Remove the keystone unit with VIP

  juju remove-unit keystone/28

After that, the cluster status become "blcoked"

Unit Workload Agent Machine Public address Ports Message
keystone/29 blocked idle 122 10.5.2.26 5000/tcp Database not initialised
  keystone-hacluster/18 active idle 10.5.2.26 Unit is ready and clustered
  keystone-mysql-router/50 active idle 10.5.2.26 Unit is ready
keystone/30* blocked idle 123 10.5.3.123 5000/tcp Database not initialised
  keystone-hacluster/17 active idle 10.5.3.123 Unit is ready and clustered
  keystone-mysql-router/49* active idle 10.5.3.123 Unit is ready
keystone/31 blocked idle 124 10.5.2.152 5000/tcp Database not initialised
  keystone-hacluster/19* active idle 10.5.2.152 Unit is ready and clustered
  keystone-mysql-router/51 active idle 10.5.2.152 Unit is ready

Detail Information
=================

I find related log in keystone/29.

2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Checking for maintenance notifications
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 This unit (keystone/29) is in allowed unit list from keystone-mysql-router/50
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 db-initialised key missing, assuming db is not initialised
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Database initialised: False
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Database not initialised
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Database not initialised
2021-09-01 03:28:42 INFO unit.keystone/29.juju-log server.go:314 Keystone charm unit not ready - deferring identity-relation updates
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Checking for maintenance notifications
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 This unit (keystone/29) is in allowed unit list from keystone-mysql-router/50
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 db-initialised key missing, assuming db is not initialised
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Database initialised: False
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Database not initialised
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Telling peers this unit is: NOTREADY
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Telling peer behind relation cluster:311 that keystone/29 is NOTREADY

At the meantime, one of the mysql-innodb-cluster unit become error.
This error can be fixed after this command

   juju resolve --no-retry mysql-innodb-cluster/4

But keystone is still stay in blocked status.

Unit Workload Agent Machine Public address Ports Message
mysql-innodb-cluster/3 active idle 69 10.5.3.213 Unit is ready: Mode: R/W, Cluster is ONLIN
E and can tolerate up to ONE failure. mysql-innodb-cluster/4 error idle 70 10.5.1.215 hook failed: "update-status"
mysql-innodb-cluster/5* active idle 71 10.5.1.138 Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.

I will attach juju crashdump later, it's too big to upload now.

Expected result
===============================
keystone cluster action complete and the unit is "active" state.