k8s charm upgrade stuck on downloading charm (2.9.44)

Bug #2038821 reported by Barry Price
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
New
Undecided
Unassigned

Bug Description

Hi

Attempting to upgrade from ch:amd64/focal/mattermost-k8s-25 to ch:amd64/focal/mattermost-k8s-26 resulted in the application status being stuck at "downloading charm":

mattermost .../mattermost-canonical-fl... maintenance 2 mattermost-k8s stable 26 10.x.y.z no downloading charm (ch:amd64/focal/mattermost-k8s-26)

After waiting for several minutes with no progress, I manually edited the deployment with kubectl to force new pods to be spawned, which seemed to "unstick" juju.

Via debug-log at the time of the upgrade:

application-mattermost: 06:24:55 INFO juju.worker.caasoperator.uniter.mattermost/16.charm downloading ch:amd64/focal/mattermost-k8s-26 from API server
application-mattermost: 06:24:55 INFO juju.downloader downloading from ch:amd64/focal/mattermost-k8s-26
application-mattermost: 06:24:55 INFO juju.worker.caasoperator.uniter.mattermost/15.charm downloading ch:amd64/focal/mattermost-k8s-26 from API server
application-mattermost: 06:24:55 INFO juju.downloader downloading from ch:amd64/focal/mattermost-k8s-26
application-mattermost: 06:24:55 INFO juju.worker.caasoperator.charm downloading ch:amd64/focal/mattermost-k8s-26 from API server
application-mattermost: 06:24:55 INFO juju.downloader downloading from ch:amd64/focal/mattermost-k8s-26
application-mattermost: 06:24:56 INFO juju.downloader download complete ("ch:amd64/focal/mattermost-k8s-26")
application-mattermost: 06:24:56 INFO juju.downloader download complete ("ch:amd64/focal/mattermost-k8s-26")
application-mattermost: 06:24:56 INFO juju.downloader download verified ("ch:amd64/focal/mattermost-k8s-26")
application-mattermost: 06:24:56 INFO juju.downloader download complete ("ch:amd64/focal/mattermost-k8s-26")
application-mattermost: 06:24:56 INFO juju.downloader download verified ("ch:amd64/focal/mattermost-k8s-26")
application-mattermost: 06:24:56 INFO juju.downloader download verified ("ch:amd64/focal/mattermost-k8s-26")
application-mattermost: 06:48:58 INFO juju.worker.caasoperator.uniter.mattermost/16 found queued "upgrade-charm" hook
application-mattermost: 06:48:59 INFO juju.worker.caasoperator.runner start "mattermost/17"

As you can see, nothing happened for about 24 minutes after the charm was apparently downloaded and verified, which lines up with the time I edited the deployment to modify an environment variable in order to force new pods.

Once I'd done that, everything proceeded as expected.

The behaviour is reminiscent of LP:1813884, but that was addressed years ago.

Revision history for this message
Ian Booth (wallyworld) wrote :

If this happens again it would be good to get a juju dump-db with yaml snippets for the affected application, unit, unittates documents.

tags: added: upgrade-charm
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.