The error 'inappropriate relation-changed for <unit>: unit has not joined' during upgrade

Just the unit gnocchi/2 has a failed state during upgrading juju controllers and models to 2.8.1, other units are all in active state.

$ juju status | grep -E "error|block|wait|exec|lost|down|fail"
gnocchi 4.2.5 error 2/3 gnocchi jujucharms 30 ubuntu
gnocchi/2 active failed 11/lxd/4 10.139.108.126 8041/tcp

However, we indeed see the flag run-default-upgrade-charm from the log, so upgrade-charm has been triggered according to [1].

The customer said they just ran 'juju upgrade-juju' to upgrade juju controllers and models, they didn't run 'upgrade-charm'. I'm not sure if juju upgrade-juju' will trigger upgrade-charm hook, so I did a test, which shows 'juju upgrade-juju' will not trigger upgrade-charm hook. Who triggered upgrade-charm is a mystery to me now.

We put aside who triggered upgrade-charm and went on to find two possible clues from the log:

1, the error 'TypeError: expected str, bytes or os.PathLike object, not NoneType'

2, lots of error 'inappropriate relation-changed for <unit>: unit has not joined'

2020-07-27 06:01:36 ERROR juju.worker.uniter agent.go:31 resolver loop error: preparing operation "run relation-changed (78; unit: mysql/2) hook": inappropriate "relation-changed" for "mysql/2": unit has not joined
2020-07-27 06:01:36 ERROR juju.worker.dependency engine.go:671 "uniter" manifold worker returned unexpected error: preparing operation "run relation-changed (78; unit: mysql/2) hook": inappropriate "relation-changed" for "mysql/2": unit has not joined

Above error 1) is gone now after applying for one patch [2] by hand and run the following two commands, but gnocchi/2 is still in a failed state.

juju run --unit gnocchi/2 -- 'charms.reactive -p clear_flag run-default-upgrade-charm ; charms.reactive -p clear_flag ceph.create_pool.req.sent ; hooks/update-status'
juju run --unit gnocchi/2 -- 'hooks/shared-db-relation-changed'

For above error 2), wallyworld told me juju 2.8.2 can fix the problem, it could be related to the change in how unit state is stored, 2.8.2 includes code changes necessary to address the hook execution bug.

My question is that even if 2.8.2 can fix the problem, but it was caused during upgrading controllers and models to 2.8.1, how can we continue to upgrade controllers and models from 2.8.1 to 2.8.2 when gnocchi/2 is still in a failed state, is there a manual step that needs to be done to fix already upgraded systems?

We also checked the log of mysql/2

2020-07-22 12:23:36 INFO juju-log Unit is ready
2020-07-22 12:23:53 ERROR juju.worker.dependency engine.go:671 "leadership-tracker" manifold worker returned unexpected error: error while mysql/2 waiting for mysql leadership release: error blocking on leadership release: lease manager stopped
2020-07-22 12:23:53 ERROR juju.worker.dependency engine.go:671 "log-sender" manifold worker returned unexpected error: cannot send log message: tls: use of closed connection
2020-07-22 12:23:53 ERROR juju.worker.uniter agent.go:31 resolver loop error: could not acquire lock: cancelled acquiring mutex
2020-07-22 12:23:53 ERROR juju.worker.dependency engine.go:671 "migration-minion" manifold worker returned unexpected error: watcher has been stopped (stopped)
2020-07-22 12:23:53 ERROR juju.worker.dependency engine.go:671 "migration-inactive-flag" manifold worker returned unexpected error: watcher has been stopped (stopped)
2020-07-22 12:23:53 ERROR juju.worker.dependency engine.go:671 "api-caller" manifold worker returned unexpected error: api connection broken unexpectedly
2020-07-22 12:23:53 ERROR juju.worker.uniter agent.go:34 updating agent status: connection is shut down
2020-07-22 12:23:57 ERROR juju.worker.dependency engine.go:671 "api-caller" manifold worker returned unexpected error: [f2b55b] "unit-mysql-2" cannot open api: unable to connect to API: dial tcp 10.139.108.184:17070: connect: connection refused
2020-07-22 12:24:01 ERROR juju.worker.dependency engine.go:671 "api-caller" manifold worker returned unexpected error: [f2b55b] "unit-mysql-2" cannot open api: unable to connect to API: dial tcp 10.139.108.147:17070: connect: connection refused

and checked the relation between gnocchi/2 and mysql/2

$ juju run -u gnocchi/2 -- relation-ids shared-db
shared-db:78
$ juju run -u gnocchi/2 -- relation-list -r shared-db:78
Nothing replied.

That's all the context history and logs I have of the case, if you need anthing else, pls let me know. thanks.

[1] https://gitlab.global.garrservices.it/cloud/charms/gnocchi/blob/abffaa2578b701a72fece33c51cb2cf02d54916d/reactive/layer_openstack.py#L42
[2] https://review.opendev.org/#/c/716545/4/src/reactive/gnocchi_handlers.py
[3] https://discourse.juju.is/t/wip-juju-2-8-2-release-notes/3396