Comment 2 for bug 1980114

There is nothing in `juju debug-log`.

I think I came across another variation of this when scaling down from 2 units to zero: status is stuck on unknown/lost (terminated) forever, even though kubectl reports no units:

$ kubectl describe po prometheus-1 -n welcome

Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal Killing 4m21s kubelet Stopping container charm
  Normal Killing 4m21s kubelet Stopping container prometheus
  Warning Unhealthy 4m21s kubelet Readiness probe failed: Get "http://10.1.16.214:38813/v1/health?level=ready": dial tcp 10.1.16.214:38813: connect: connection refused
  Warning Unhealthy 4m21s kubelet Liveness probe failed: Get "http://10.1.16.214:38813/v1/health?level=alive": dial tcp 10.1.16.214:38813: connect: connection refused

... after 5 min ...

$ kubectl describe po prometheus-0 -n welcome
Error from server (NotFound): pods "prometheus-0" not found

$ kubectl describe po prometheus-1 -n welcome
Error from server (NotFound): pods "prometheus-1" not found

$ juju status

Model Controller Cloud/Region Version SLA Timestamp
welcome chdv32 microk8s/localhost 2.9.32 unsupported 10:03:56-04:00

App Version Status Scale Charm Channel Rev Address Exposed Message
prometheus terminated 0 prometheus-k8s 0 10.152.183.138 no unit stopped by the cloud

Unit Workload Agent Address Ports Message
prometheus/0 unknown lost 10.1.16.226 agent lost, see 'juju show-status-log prometheus/0'
prometheus/1 unknown lost 10.1.16.214 agent lost, see 'juju show-status-log prometheus/1'

Relation provider Requirer Interface Type Message
prometheus:prometheus-peers prometheus:prometheus-peers prometheus_peers peer