Keystone become "Database not initialised" after scale up and scale down

Attach the juju crashdump

Other people remind me that I can add --small when doing crashdump.
Attach the crashdump again.

I get this crashdump after the mysql innodb cluster recovered.

Thanks a lot for reporting Eric! Indeed I confirm that I have seen it myself. This is in fact visible and worked around in one of our test gates:

https://github.com/openstack-charmers/zaza-openstack-tests/blob/master/zaza/openstack/charm_tests/hacluster/tests.py#L154

So by running this test gate the issue can easily be reproduced.

I'm suspecting this is caused by this particular reactive handler:

@hooks.hook('shared-db-relation-departed',
            'shared-db-relation-broken')
def db_departed_or_broken():
    if is_leader():
        leader_set({'db-initialised': None})

Perhaps if you remove the current leader of the keystone app, this would be triggered?

And should perhaps juju move leadership away from the node to be removed before firing any hook?

BootStack encountered this same error today, when removing and re-adding a keystone unit on a cloud.

the cloud is running focal/ussuri, and keystone is on cs:keystone-330 . In our case, mysql did not go into an error state, but we were able to work around the issue by setting the db-initialised flag:

$ juju run -u keystone/leader 'leader-set db-initialised=True'

Note that we had verified keystone itself was running as expected, but relational data was not being shared (nor relations created) with sibling units, via the identity-service interface.

Added Juju, to determine if the leadership should be removed prior to a unit being removed.

Given the significant impact on production clouds when this bug occurs, I would like to see us re-evaluate the importance setting.

Juju is designed to keep a unit the leader during the dying phase and will only revoke leadership once the unit transitions to dead, just prior to removal and after all hooks have been run.

The unit, if leader prior to removal, retains leadership so that it may do "leader" type things during the relation departed/broken hooks.

I'll mark as Invalid for juju since the leadership behaviour is by design.

> Juju is designed to keep a unit the leader during the dying phase and will only revoke leadership once the unit transitions to dead, just prior to removal and after all hooks have been run.

> The unit, if leader prior to removal, retains leadership so that it may do "leader" type things during the relation departed/broken hooks.

Just seeking clarification, but essentially, in a multi-unit set-up, if the leader is removed, a leadership election doesn't take place until the unit is removed?

So to pose a question: how does a unit 'know' that, when it is dying, that it isn't the last unit and not to do things like clean-up, etc? i.e. as the leader, how should it determine not to do clean-up? Is there an (option of/way to implement) atomic counter of units that are still 'alive' in the application?

If you need to know about counterpart units, that is what a peer relation is for. If you want to know if your unit is going away that is $JUJU_DEPARTING_UNIT, IIRC.

> If you need to know about counterpart units, that is what a peer relation is for. If you want to know if your unit is going away that is $JUJU_DEPARTING_UNIT, IIRC.

Has the behaviour in juju changed at all since earlier versions? e.g. around departing units and leadership. It's just keystone has been around for a long time, and it's curious that this bug has surfaced. Certainly, it has a bug, but I'm wondering if leadership was dropped on the departing unit unless it was the last one, previously?

I just encountered this exact same problem while shuffling my units around compute hosts.

I am using Focal/Xena with keystone charm revision 576 from the xena/stable channel.

- Added a new replacement unit to a new node going up from 3 keystone to 4

- Juju remove-unit keystone/1* (leader)

- All keystone units started displaying the "Database not initialised" message.

- Executing the following command resolved the issue:

$ juju run -u keystone/leader 'leader-set db-initialised=True'

Okay, so the reason this is happening is due to this code in keystone:

hooks/keystone_hooks.py, line 442:

@hooks.hook('shared-db-relation-departed',
            'shared-db-relation-broken')
def db_departed_or_broken():
    if is_leader():
        leader_set({'db-initialised': None})

In order to fix it, the code should verify (via the cluster/peer relationship) whether it is the only unit and NOT set the flag 'db-initialised' to None. However, if it's the only unit left, it would be valid to clear it.

Hi everyone,

I thought this was an interesting issue to pick up.

I initially implemented @ajkavanagh 's proposal but someone pointed out to me that the 'shared-db-relation-departed' and 'shared-db-relation-broken' hooks are also ran if I run ""juju remove-relation keystone:shared-db mysql-router", so the problem became :

- The charm should NOT set the flag 'db-initialised' to None if I run "juju remove-unit" on the leader unit
when it's not the only unit
- But the charm should also correctly set the flag to None if I run "juju remove-relation"
- All that has to be done in the hook callback without knowing the whole context (which command triggered the hook, is the unit getting torn down or is it a simple remove-relation)

So I decided to set a new leader value and next time db-initialised is checked, the leader (not necessarily the unit that set this value to None) can use this extra information to revert if needed. I have a patch for this here : https://review.opendev.org/c/openstack/charm-keystone/+/897259.

Can someone have a look and let me know what they think?

Thanks!
Thanh