vault fails to start when MySQL backend down

This can be reproduced with cs:vault-35 :

1. Deploy, initialize and unseal a vault:

juju deploy cs:vault-35
juju deploy mysql
juju relate vault mysql
export VAULT_ADDR="http://$(juju run --unit vault/0 'unit-get private-address'):8200"
vault operator init -key-shares=5 -key-threshold=2
vault operator unseal A1I4gVtqqFoDBEoQznosX+kmCnFRqOiNhq4Xq5GZtR9y
vault operator unseal K/ASAolWEA1ngDidJ+yEsUP1q6mK4I5tK2GRH+RrRQsv
export VAULT_TOKEN=s.sNyy3wrtrNoVUDj0NCWmMyUd
vault token create -ttl=10m
juju run-action --wait vault/0 authorize-charm token=s.vNrCZl8c9qUXueQhgqrr0kQP

2. Pause the vault and stop mysql:

juju run-action --wait vault/0 pause # -> blocked: Vault service not running
juju run --unit mysql/0 -- systemctl stop mysql

3. Resume the vault:

juju run-action --wait vault/0 resume # -> still blocked: Vault service not running

Logs show:

2020-03-09 14:34:17 ERROR juju-log Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.6/site-packages/urllib3/connection.py", line 157, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

4. Resurrect mysql:

juju run --unit mysql/0 -- systemctl start mysql

Expected: the vault service resurrects at some point
Actual: the vault service remains stopped until the operator performs a 'resume' action again.

It is also possible to reach "blocked: Vault health check failed" instead of "blocked: Vault service not running"

In the case of "blocked: Vault health check failed", the vault service is still running and able to recover when MySQL gets back up. In the case of "blocked: Vault service not running", it won't recover on its own but systemd has detected the service as failed, so indeed it should be possible to solve the issue by tuning Restart*= on the systemd unit.

I filed Bug #1871539 to handle the problem where the charm gets stuck in the blocked state, and you cannot get out of it (even with the resume action or any other juju incantation) - due to faults in the flag logic.

This bug can potentially deal with the general problem of not retrying for MySQL for a while.