<name>_relation_departed is sometimes called randomly or skipped

Certainly randomly running relation-departed when a unit hasn't actually been removed (and then not running it when the relation *is* removed) is a critical function. I can't say we've seen this elsewhere, but I'll dig and see if we can reproduce

$ juju model-config logging-config
<root>=WARNING;unit=DEBUG

so one bit that doesn't quite fit in the picture. Looking at https-client_34/var/log/juju/machine-lock.log we see:
2021-09-29 16:31:28 === agent unit-https-client-34 started ===
2021-09-29 16:31:28 unit-https-client-34: meterstatus (meter-status-changed), waited 0s, held 0s
2021-09-29 16:31:33 unit-https-client-34: https-client/34 uniter (run install hook), waited 0s, held 1s
2021-09-29 16:31:33 unit-https-client-34: https-client/34 uniter (run relation-created (39; app: lxd) hook), waited 0s, held 1s
2021-09-29 16:31:34 unit-https-client-34: https-client/34 uniter (run leader-settings-changed hook), waited 0s, held 1s
2021-09-29 16:31:35 unit-https-client-34: https-client/34 uniter (run config-changed hook), waited 0s, held 1s
2021-09-29 16:31:35 unit-https-client-34: https-client/34 uniter (run start hook), waited 0s, held 1s
2021-09-29 16:35:48 unit-https-client-34: https-client/34 uniter (run update-status hook), waited 0s, held 1s
2021-09-29 16:35:51 unit-https-client-34: https-client/34 uniter (run relation-broken (39; app: lxd) hook), waited 0s, held 1s
2021-09-29 16:40:19 unit-https-client-34: https-client/34 uniter (run update-status hook), waited 0s, held 1s
2021-09-29 16:46:02 unit-https-client-34: https-client/34 uniter (run update-status hook), waited 0s, held 1s
2021-09-29 16:50:32 unit-https-client-34: https-client/34 uniter (run update-status hook), waited 0s, held 1s
2021-09-29 16:56:10 unit-https-client-34: https-client/34 uniter (run update-status hook), waited 0s, held 1s
2021-09-29 17:01:01 unit-https-client-34: https-client/34 uniter (run update-status hook), waited 0s, held 1s
2021-09-29 17:06:29 unit-https-client-34: https-client/34 uniter (run update-status hook), waited 0s, held 1s
2021-09-29 17:10:47 unit-https-client-34: https-client/34 uniter (run update-status hook), waited 0s, held 1s
2021-09-29 17:16:28 unit-https-client-34: https-client/34 uniter (run update-status hook), waited 0s, held 1s
2021-09-29 17:22:22 unit-https-client-34: https-client/34 uniter (run update-status hook), waited 0s, held 1s
2021-09-29 17:27:54 unit-https-client-34: https-client/34 uniter (run update-status hook), waited 0s, held 1s
2021-09-29 17:32:06 unit-https-client-34: https-client/34 uniter (run update-status hook), waited 0s, held 1s

That clearly points to it running relation-created, start and update-status, but never running relation-joined, and never running relation-departed.
I *would* expect that if a unit doesn't join a relation then it won't depart the relation, and thus you would get broken without departed.

however, I just noticed that the original bug was about unit 33, will check there.

2021-09-29 16:30:50 === agent unit-https-client-33 started ===
2021-09-29 16:30:51 unit-https-client-33: meterstatus (meter-status-changed), waited 0s, held 0s
2021-09-29 16:30:56 unit-https-client-33: https-client/33 uniter (run install hook), waited 0s, held 1s
2021-09-29 16:30:56 unit-https-client-33: https-client/33 uniter (run leader-elected hook), waited 0s, held 1s
2021-09-29 16:30:57 unit-https-client-33: https-client/33 uniter (run config-changed hook), waited 0s, held 1s
2021-09-29 16:30:58 unit-https-client-33: https-client/33 uniter (run start hook), waited 0s, held 1s
2021-09-29 16:31:10 unit-https-client-33: https-client/33 uniter (run relation-created (39; app: lxd) hook), waited 0s, held 1s
2021-09-29 16:31:11 unit-https-client-33: https-client/33 uniter (run relation-joined (39; unit: lxd/21) hook), waited 0s, held 1s
2021-09-29 16:31:12 unit-https-client-33: https-client/33 uniter (run relation-changed (39; unit: lxd/21) hook), waited 0s, held 1s
2021-09-29 16:31:13 unit-https-client-33: https-client/33 uniter (run relation-changed (39; unit: lxd/21) hook), waited 0s, held 1s
2021-09-29 16:31:29 unit-https-client-33: https-client/33 uniter (run relation-departed (39; unit: lxd/21, departee: lxd/21) hook), waited 0s, held 1s
2021-09-29 16:35:51 unit-https-client-33: https-client/33 uniter (run relation-broken (39; app: lxd) hook), waited 0s, held 1s
2021-09-29 16:35:54 unit-https-client-33: https-client/33 uniter (run update-status hook), waited 0s, held 1s
...

Note that this correlates with:
lxd_21/var/log/juju/machine-lock.log
2021-09-29 16:30:25 unit-lxd-21: lxd/21 uniter (run relation-departed (38; unit: https-client/30, departee: https-client/30) hook), waited 0s, held 1s
2021-09-29 16:30:26 unit-lxd-21: lxd/21 uniter (run relation-departed (38; unit: https-client/31, departee: https-client/31) hook), waited 0s, held 1s
2021-09-29 16:30:27 unit-lxd-21: lxd/21 uniter (run relation-departed (38; unit: https-client/32, departee: https-client/32) hook), waited 0s, held 1s
2021-09-29 16:30:28 unit-lxd-21: lxd/21 uniter (run relation-broken (38; app: https-client) hook), waited 0s, held 1s
2021-09-29 16:31:10 unit-lxd-21: lxd/21 uniter (run relation-created (39; app: https-client) hook), waited 0s, held 1s
2021-09-29 16:31:11 unit-lxd-21: lxd/21 uniter (run relation-joined (39; unit: https-client/33) hook), waited 0s, held 1s
2021-09-29 16:31:13 unit-lxd-21: lxd/21 uniter (run relation-changed (39; unit: https-client/33) hook), waited 0s, held 1s
2021-09-29 16:31:22 unit-lxd-21: lxd/21 uniter (run update-status hook), waited 0s, held 1s
2021-09-29 16:31:30 unit-lxd-21: lxd/21 uniter (run relation-departed (39; unit: https-client/33, departee: https-client/33) hook), waited 0s, held 1s
2021-09-29 16:31:34 unit-lxd-21: lxd/21 uniter (run relation-joined (39; unit: https-client/34) hook), waited 0s, held 1s
2021-09-29 16:31:35 unit-lxd-21: lxd/21 uniter (run relation-changed (39; unit: https-client/34) hook), waited 0s, held 1s
2021-09-29 16:33:53 unit-lxd-21: lxd/21 uniter (run relation-joined (39; unit: https-client/35) hook), waited 0s, held 1s
2021-09-29 16:33:54 unit-lxd-21: lxd/21 uniter (run relation-changed (39; unit: ht...

So what this *appears* to be is that somehow we ended up with the 'juju add-unit https-client' suddenly treating the relation as having no units in the relation (which is why https-client/33 shows departed being called, and lxd/21 departs, and then why 34 and 35 show -created but never -joined or -changed, and why they then only show -broken and never -departed.)

I don't have any idea how Juju would see an established relation, and then remove the active units due to an add-unit.

This may be a similar cause to:
https://bugs.launchpad.net/bugs/1940983

though in that case cross model relations were involved.

To give a rough timeline:
lxd/21 is up and running
https-client/30, 31, 32 have just gone away

2021-09-29 16:30:27 unit-lxd-21: lxd/21 uniter (run relation-departed (38; unit: https-client/32, departee: https-client/32) hook), waited 0s, held 1s
2021-09-29 16:30:28 unit-lxd-21: lxd/21 uniter (run relation-broken (38; app: https-client) hook), waited 0s, held 1s

https-client/33 shows up and joins the relation:
2021-09-29 16:31:10 unit-lxd-21: lxd/21 uniter (run relation-created (39; app: https-client) hook), waited 0s, held 1s
2021-09-29 16:31:11 unit-lxd-21: lxd/21 uniter (run relation-joined (39; unit: https-client/33) hook), waited 0s, held 1s
2021-09-29 16:31:13 unit-lxd-21: lxd/21 uniter (run relation-changed (39; unit: https-client/33) hook), waited 0s, held 1s
2021-09-29 16:31:22 unit-lxd-21: lxd/21 uniter (run update-status hook), waited 0s, held 1s

https-client/34 and 35 are added, somehow removing 33:
2021-09-29 16:31:30 unit-lxd-21: lxd/21 uniter (run relation-departed (39; unit: https-client/33, departee: https-client/33) hook), waited 0s, held 1s
2021-09-29 16:31:34 unit-lxd-21: lxd/21 uniter (run relation-joined (39; unit: https-client/34) hook), waited 0s, held 1s
2021-09-29 16:31:35 unit-lxd-21: lxd/21 uniter (run relation-changed (39; unit: https-client/34) hook), waited 0s, held 1s
2021-09-29 16:33:53 unit-lxd-21: lxd/21 uniter (run relation-joined (39; unit: https-client/35) hook), waited 0s, held 1s
2021-09-29 16:33:54 unit-lxd-21: lxd/21 uniter (run relation-changed (39; unit: https-client/35) hook), waited 0s, held 1s
2021-09-29 16:35:37 unit-lxd-21: lxd/21 uniter (run update-status hook), waited 0s, held 1s
2021-09-29 16:35:51 unit-lxd-21: lxd/21 uniter (run relation-departed (39; unit: https-client/34, departee: https-client/34) hook), waited 0s, held 1s

but also in some sort of fashion that 34 and 35 never see lxd/21
2021-09-29 16:31:35 unit-https-client-34: https-client/34 uniter (run start hook), waited 0s, held 1s
2021-09-29 16:35:48 unit-https-client-34: https-client/34 uniter (run update-status hook), waited 0s, held 1s
2021-09-29 16:35:51 unit-https-client-34: https-client/34 uniter (run relation-broken (39; app: lxd) hook), waited 0s, held 1s

This may also be the case that was seen in
https://bugs.launchpad.net/juju/+bug/1931632

Ideally we'd turn on TRACE debugging for a couple of key modules so we can see in more detail what's happening when it goes wrong:

juju.state.relationunits
juju.worker.uniter.relation
juju.worker.uniter.relationunits

And also get a "juju dump-db" which produces a yaml db dump.
(you'd need to set JUJU_DEV_FEATURE_FLAGS=developer-mode on the client to get this command)

@wallyworld, thanks. I made the following change and will provide the DB dump if/when I run into the issue again.

$ juju model-config logging-config
<root>=WARNING;unit=DEBUG;juju.state.relationunits=TRACE;juju.worker.uniter.relation=TRACE;juju.worker.uniter.relationunits=TRACE

Adding to a milestone so we can track the bug; we can better investigate once we get the dump-db output.

I'll mark as incomplete until we get told of another case where it's happened again and we can get the necessary debug info.

[Expired for Canonical Juju because there has been no activity for 60 days.]