ocn rev 105 Unable to authorize approle after unseal

The following introduced a gate health check, client_approle_authorized, to handle database topology changes (rolling restarts, pause/resumes, etc). It checks that the local charm can authorize itself. A tenacity retry is also added.
https://review.opendev.org/#/c/740086/
https://review.opendev.org/#/c/739129/

These changes have caused delays during long update-status hook executions.

As it turns out SQA uses a 5 minute TTL on their token create:
juju run -u vault/leader 'export VAULT_TOKEN=<token> && export VAULT_ADDR=http://127.0.0.1:8200 && /snap/bin/vault token create --ttl=5m'

If an update-status is running ahead of the above juju run we get a time out on the juju run.
If update-status runs after this but before the action, authorize-charm is run we get the following "permission denied" error because the token TTL has been exceeded.

2020-08-02 06:18:05 ERROR juju-log Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vault-0/charm/actions/authorize-charm", line 185, in main
    action(args)
  File "/var/lib/juju/agents/unit-vault-0/charm/actions/authorize-charm", line 45, in authorize_charm_action
    role_id = vault.setup_charm_vault_access(action_config['token'])
  File "/var/lib/juju/agents/unit-vault-0/charm/lib/charm/vault.py", line 213, in setup_charm_vault_access
    enable_approle_auth(client)
  File "/var/lib/juju/agents/unit-vault-0/charm/lib/charm/vault.py", line 178, in enable_approle_auth
    if 'approle/' not in client.list_auth_backends():
  File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.6/site-packages/hvac/v1/__init__.py", line 1738, in list_auth_backends
    return self._adapter.get('/v1/sys/auth').json()
  File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 90, in get
    return self.request('get', url, **kwargs)
  File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 233, in request
    utils.raise_for_error(response.status_code, text, errors=errors)
  File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.6/site-packages/hvac/utils.py", line 33, in raise_for_error
    raise exceptions.Forbidden(message, errors=errors)
hvac.exceptions.Forbidden: permission denied

Root cause:
The gate, client_approle_authorized, checking for app role authorization is called before the charm has been authorized causing tenacity retries and long update-state hook executions ultimately exceeding the 5 minute token TTL.

TRIAGE:
Add a check in client_approle_authorized for leader setting of local-charm-access-id which is set during the authorize-charm action.

My bug fix: https://review.opendev.org/#/c/744536/

Alex and I had a further discussion about the vault charm that I will summarize here. The changes mentioned in comment #1 were necessary because the charm runs state changing functions during update-status hook executions. Another approach might be to revert the original changes and stop the charm from executing these functions during an update-status hook. This might also be a good 20.10 goal for the charm.

This [0] has landed and needs to be verified by SQA.

[0] https://review.opendev.org/#/c/744536/