When new revisions for the kubectl snap are released we are seeing this error in the k-c-p logs as well as the unit being in error state:
2022-06-19 23:57:18 INFO unit.kubernetes-master/5.juju-log server.go:327 coordinator:11: status-set: maintenance: Joining snap cohort.
2022-06-19 23:57:18 WARNING unit.kubernetes-master/5.coordinator-relation-changed logger.go:60 error: cannot refresh "kubectl": snap "kubectl" has running apps (kubectl)
2022-06-19 23:57:18 ERROR unit.kubernetes-master/5.juju-log server.go:327 coordinator:11: Hook error:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-kubernetes-master-5/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 74, in main
bus.dispatch(restricted=restricted_mode)
File "/var/lib/juju/agents/unit-kubernetes-master-5/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
_invoke(other_handlers)
File "/var/lib/juju/agents/unit-kubernetes-master-5/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
handler.invoke()
File "/var/lib/juju/agents/unit-kubernetes-master-5/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
self._action(*args)
File "/var/lib/juju/agents/unit-kubernetes-master-5/charm/reactive/kubernetes_master.py", line 469, in join_or_update_cohorts
snap.join_cohort_snapshot(snapname, cohort_key)
File "lib/charms/layer/snap.py", line 455, in join_cohort_snapshot
subprocess.check_output(["snap", "refresh", snapname, "--cohort", cohort_key])
File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
**kwargs).stdout
File "/usr/lib/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['snap', 'refresh', 'kubectl', '--cohort', 'REDACTED']' returned non-zero exit status 1.
I believe that this is similar to LP#1978005 [1] and LP#1975714 [2]. The fix for this is currently in version 2.57, which is in candidate. Once this is released, AIUI the kubectl snap would need to be changed to set `refresh-mode: ignore-running` in order for refreshes to be processed [3][4].
[1] https://bugs.launchpad.net/snapd/+bug/1978005
[2] https://bugs.launchpad.net/snapd/+bug/1975714
[3] https://github.com/snapcore/snapd/pull/11855/files
[4] https://chat.canonical.com/canonical/pl/47m58wxjt3bzfehpedogg7dwio
I'm able to repro by running a snap refresh to a different revision while running kubectl in the background.
The documentation around `refresh-mode: ignore-running` suggests that it's for daemons[1], but I'll give it a try and see if it works for non-daemon apps too.
[1]: https:/