Juju does not surface errors polling CMR offers on other controllers

In other words, to repro:

(1) create a LXD controller
(2) create a microk8s controller
(3) deploy something in microk8s that is offered to and consumed by the LXD controller
(4) nuke the microk8s controller and spin up another in its place
(5) see the LXD controller get really, really confused

I've tried to reproduce this on 2.9.19 (the version suggested by the bug description) and also 2.9.22 (a cmr worker shutdown bug 1952796 was fixed in 2.9.22).

On 2.9.22, on microk8s, I deployed and offered charmed-osm-mariadb-k8s.
On a LXD controller, I deployed mediawiki and related to the k8s mariadb offer.

I then ran juju kill-controller on the microk8s controller. juju status on the consuming model showed the mariadb SAAS as "terminated" and then shortly after it was removed from status. The logs showed the network related "server misbehavng" error (not repeated).

I bootstrapped a new microk8s controller with the same name as before and recreated the offer and added the cross model relation again and things worked as expected.

On 2.9.19, killing the controller resulted in the mariadb SAAS showing as "terminated" but it was not removed. I recreated the microk8s controller and this time needed to remove the terminated SAAS, after which I could relate mediawiki to the k8s mariadb offer and things worked as expected.

I am doing this on a single host so the replacement microk8s controller does have the same IP address as the original one. Perhaps that's why it works for me.

Can you confirm whether the replacement microk8s controller in your scenario was created on a different host?

It might detect the SAAS offer being terminated because you do kill the controller. I don’t: I uninstall the juju and microk8s snaps directly. I realize this is kinda a “saboteur” test, but we should not assume Juju controllers are around to be killed in case of infrastructure failure.

And, no, I always create the controllers on the same host. However, the IP of MicroK8s changes because I flip-flop a bunch between networks at home and at the office.

I can reproduce the behaviour using 2 LXD controllers, killing the one with the offer.

When the consuming side connects to an offering controller and an error occurs, the worker which hosts that functionality will restart because it's hard to tell if a given error is transient - the restart will cause a retry. Especially when the root cause of an error is not an internal juju issue but something network related, the best juju can do is log the fact that the API connection to the target controller could not be opened.

What juju does not do but should is mark the SAAS application as in error which needs to be fixed.

In response, what the user can do is "juju remove-saas foo --force". It can be expected that the log spam would then stop as the need to contact the target controller is now redundant since the SAAS application is gone. I think though there's a bug in that doing this might now stop the polling of the controller so that should be fixed.

This PR will show the SAAS error if the offering controller is stopped / removed.
With LXD, I stopped the controller instance, saw the error. Then I started the instance again and SAAS status went back to active.

https://github.com/juju/juju/pull/13637

There's a fair bit more work to do to add the ref counting needed to shutdown polling of the offering controller if it goes away. That will need to be a separate PR.

I'll split this bug in 2 as separate PRs will be done for each issue.
This bug: as per the above PR, status will show any error polling the target controller.
Additional bug: juju does not stop polling a controller even when all offers on that controller are no longer being consumed.

This is the second bug raised https://bugs.launchpad.net/juju/+bug/1958446