Unable to remove an offer, claiming removed relation still exists

juju destroy-model hangs, and there is no --force option, so cleanup requires destroying the controller.

This model may have been stuck due to https://bugs.launchpad.net/juju/+bug/1813884, but the error message is still clearly incorrect and indicates inconsistent state.

(manual destruction of the controller, since destroy-controller --destroy-all-models also hangs and also has no --force option)

Do you know what 'juju offers' run on hte offering model printed?
This will show if Juju thinks there are any consumers still using the offer.

I can reproduce this.

$ juju offers
Offer User Relation id Status Endpoint Interface Role Ingress subnets
directory-db admin 0 joined postgres pgsql requirer

Full log of reproduction: https://pastebin.ubuntu.com/p/s2cW8jdH5w/

I tried to reproduce in a similar way using these charms (also using a LXD controller and microk8s):
- deploy mariadb-k8s charm to k8s model and offer it
- deploy mediawiki to IAAS model and consume the k8s offer

Everything worked as expected.

Can you give me the source to the test k8s charm you are using with postgresql and I can try your exact deployment scenario?

Discussed in IRC - it appears that the issue arises when the offering (k8s) charm has not fully come up yet and initialised its pod info. This looks like it falls into the existing class of problem where Juju is not good at removing apps/relations/units etc if there's an underlying error or the charm hasn't started running hooks yet.

I can confirm that the offer can be removed after the pod has been spun up.

This will be seen in the wild when a k8s charm blocked or waiting for information from a relation before it can usefully spin up the pod.

This is also happening with Juju 2.6.2 (currently targetted to 2.5.8).

With 2.6.2, 'juju remove-application --force' is failing when models get wedged in this state:

ubuntu@blimp:~$ juju remove-application directory --force
removing application directory failed: cannot destroy application "directory": state changing too quickly; try again soon

juju destroy-model and juju destroy-model also hang, waiting on the wedged application

juju destroy-model and juju destroy-model --force also hang, waiting on the wedged application.

I ran into this 'in the wild' in that I was attempting to add a unit that failed. Had I waited to consume the offer until after the provisioning was finished it may have been ok.

In any case, having a clean way to remove an offer without destroying the controller would make things a lot better from a user experience perspective.

The fix[0] for this bug has passed code review. It is should be part of the 2.6.5 release, which is expected to land early next week[1].

[0] https://github.com/juju/juju/pull/10334
[1] https://discourse.jujucharms.com/t/1584