Removing one ovn-central unit doesn't cluster/leave SB and NB clusters

Subscribing field-medium.

This issue can cause major outages when sb and nb clusters can't elect a leader because of stale ovn-central units in the cluster.

Starting from 3 ovn-central, we had to remove 2 ovn-central (because of hw maintenance) and added two back. We didn't manually cluster/leave. The two raft clusters were unable to elect a leader because both SB and NB had 4 members of which 2 down and the 5th unit could not join the cluster

The 3 left ovn-central were /2 /3 /4

To recreate the clusters we followed these steps:

Recovery steps:

1. stop all units:
juju run-action ovn-central/2 pause --wait
juju run-action ovn-central/3 pause --wait
juju run-action ovn-central/4 pause --wait

2. created standalone on ovn-central/2
# ovsdb-tool cluster-to-standalone /tmp/standalone_ovnsb_db.db /var/lib/ovn/ovnsb_db.db
# ovsdb-tool cluster-to-standalone /tmp/standalone_ovnnb_db.db /var/lib/ovn/ovnnb_db.db

3. create clusters
ovsdb-tool create-cluster /var/lib/ovn/ovnsb_db.db /tmp/standalone_ovnsb_db.db ssl:<ovn-central-2-ip>:6644
ovsdb-tool create-cluster /var/lib/ovn/ovnnb_db.db /tmp/standalone_ovnnb_db.db ssl:<ovn-central-2-ip>:6643

4. Resume ovn-central/2

5. Join cluster from ovn-central/3

ovsdb-tool --cid=<new-sb-cid-took-from-ovn-central-2> join-cluster /var/lib/ovn/ovnsb_db.db OVN_Southbound ssl:<ovn-central-3-ip>:6644 ssl:<ovn-central-2-ip>:6644
ovsdb-tool --cid=<new-nb-cid-took-from-ovn-central-2> join-cluster /var/lib/ovn/ovnnb_db.db OVN_Northbound ssl:<ovn-central-3-ip>:6643 ssl:<ovn-central-2-ip>:6643

6. Resuming /3
juju run-action ovn-central/3 resume --wait

7. Fixing leader-set
juju run -u ovn-central/leader leader-set nb_cid="<new-nb-cid-took-from-ovn-central-2>"
juju run -u ovn-central/leader leader-set sb_cid="<new-sb-cid-took-from-ovn-central-2>"

8. Resuming /4
juju run-action ovn-central/4 resume --wait

Definitively something the charm should handle. I think we would need to implement two entry points to cluster member removal:

    one opportunistic one running on the departing unit in its stop hook (a member departing sending a leave message is the preferred approach according to ovsdb-server documentation)

    an action which can be issued by the operator on the remaining units in the event a unit is abruptly removed from the cluster without removing itself

Proposed spec for this issue: https://review.opendev.org/c/openstack/charm-specs/+/835630

Fix proposed to branch: master
Review: https://review.opendev.org/c/x/charm-ovn-central/+/859720

Reviewed: https://review.opendev.org/c/x/charm-ovn-central/+/859720
Committed: https://opendev.org/x/charm-ovn-central/commit/34a82bc763991c7a2b6f2c23b58e3239c8058ef2
Submitter: "Zuul (22348)"
Branch: master

commit 34a82bc763991c7a2b6f2c23b58e3239c8058ef2
Author: Martin Kalcok <email address hidden>
Date: Wed Sep 28 22:34:54 2022 +0200

    Implementation of ovn-central downscaling.

    This change includes:
    * attempt to leave cluster gracefully when removing unit
    * cluster-status action that shows status of SB and NB clusters
    * cluster-kick action that allows user to remove cluster members

    Associated spec: https://opendev.org/openstack/charm-specs/src/branch/master/specs/yoga/approved/ovn-central-downscaling.rst

    Closes-Bug: #1948680
    func-test-pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/933
    Change-Id: I40ae08669d00b3b1fa567a45db2ce51425e6d1cb