Unable to upgrade k8s charm when only resource revisions changed

Bug #1993319 reported by Kian Parvin
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
Fix Released
High
Heather Lanigan
3.0
Fix Committed
High
Heather Lanigan
3.1
Fix Released
High
Heather Lanigan

Bug Description

When a k8s charm+resource is deployed, if the corresponding charm on Charmhub has a new release where only the resource revision is bumped (i.e. the charm revision remains the same), there is seemingly no way to force Juju to upgrade the OCI image in the active deployment.

It feels like a common use case to build and release new OCI images while keeping the charm revision the same, this would also seemingly relate to the k8s pull-policy.

Juju version: 2.9.35-ubuntu-amd64 (Client and controller)
Cloud: Kubernetes - Tested on microk8s

To reproduce:
- "Release" charm into a channel on charmhub with revision X for the charm and revision Y for the OCI image resource.
- Deploy the charm to k8s (tested with microk8s)
- (In my case) Tested /status endpoint which returns my Github commit hash
- "Release" the charm again, into the same channel, with revision X for the charm and revision Y+1 for the OCI image resource.
- Try `juju refresh app` and various alternatives.
- Tried restarting the pod with `microk8s kubectl get pod <pod_name> -n <namespace> -o yaml | microk8s kubectl replace --force -f -`
- Hitting the /status endpoint again confirmed the pod was not upgraded.

None of the above refresh commands or restarting the pod could get the image to upgrade.

Alternatively, if it's required that the charm revision be increased for the refresh to work then an update to the docs in https://juju.is/docs/olm/upgrade-applications would be a good alternative.

Tags: canonical-is
Revision history for this message
Ian Booth (wallyworld) wrote :

The "normal" workflow is to publish a new charm tuple containing charm and resource revision updates. The charm revision can remain the same and if there are updated resource revisions, juju is supposed to run the upgrade-charm hook.

At least that's how is used to work with the old charm store. There might be some work needed with charmhub to achieve the same behaviour.

Changed in juju:
milestone: none → 2.9.38
importance: Undecided → High
status: New → Triaged
Changed in juju:
milestone: 2.9.38 → 2.9.39
Changed in juju:
milestone: 2.9.39 → 2.9.40
Changed in juju:
assignee: nobody → Heather Lanigan (hmlanigan)
Tom Haddon (mthaddon)
tags: added: canonical-is
Changed in juju:
milestone: 2.9.40 → 2.9.41
Changed in juju:
status: Triaged → In Progress
Revision history for this message
Heather Lanigan (hmlanigan) wrote (last edit ):

I have a stepping stone pr up to handle the general case of `juju refresh` upgrading resources where there is no new charm available:

https://github.com/juju/juju/pull/15196

Next step is to ensure that ori-images are properly upgraded.

This change is unrelated to charmhub. It's not clear based on the code I found that this functionality worked with charmstore either. One of the problems resolved was due to code in state which didn't resolve pending resources unless the charm revision changed too. Nor was the CharmModifiedVersion updated to trigger the upgrade charm hook. There were more.

Revision history for this message
Heather Lanigan (hmlanigan) wrote :

With my change above, and using the loki-k8s charm, this appears to be resolved. Confirmation would be helpful.

I deployed the loki-k8s charm:

$ juju deploy loki-k8s --resource loki-image=2

Revision 2 was used for the Loki-image because it had a different SHA from the most recent resource revision.

$ microk8s kubectl get pod loki-k8s-0 -n two -o json | grep "loki-image"
                "image": "registry.jujucharms.com/charm/ayt4gaga2x20gye32qthgygevco13qgq6rfff/loki-image@sha256:4e6dca46091ac8e08f8e06e734d657ee6c0000f1727050fd70ca59af7086841d",

$ juju refresh loki-k8s
charm "loki-k8s": already up-to-date
resources to be upgraded
Leaving endpoints in "alpha": alertmanager, grafana-dashboard, grafana-source, ingress, logging, metrics-endpoint

Use `juju show-status-log loki-k8s-0` to see the upgrade-charm hook after `juju refresh` run.

$ microk8s kubectl get pod loki-k8s-0 -n two -o json | grep "loki-image"
                "image": "registry.jujucharms.com/charm/ayt4gaga2x20gye32qthgygevco13qgq6rfff/loki-image@sha256:b61a65dbf591ccebebf3874c8d5e37d4ba1a8d2017625da97a83a4ced2ff252e",

The image has a new SHA.

Revision history for this message
Tom Haddon (mthaddon) wrote :

Deploying with an older version of the published resource and then confirming `juju refresh` upgrades the resource is a bit different to deploying what's published in charmhub, and then uploading a new resource and releasing the same charm revision with the new resource revision and confirming if `juju refresh` does the right thing in this case.

Revision history for this message
Heather Lanigan (hmlanigan) wrote :

@mthaddon,

This is the closest approximation to the scenario which I can easily to reproduce. I do not have an existing deployment available where I see if the new code resolves the bug.

From juju's perspective, there is no difference in the code. Once a charm is deployed with a resource, any details about how that was done is lost unless the resource was uploaded. If a resource is uploaded you cannot upgrade it to a version of the resource in charmhub. Any difference, that I'm aware of, would be in charmhub response to the refresh/refresh call that are not within juju's control.

I'll see if I can put together a staging charm for test.

Changed in juju:
milestone: 2.9.41 → 2.9.42
Changed in juju:
milestone: 2.9.42 → 2.9.43
Revision history for this message
Heather Lanigan (hmlanigan) wrote :

I created a charm for test and put in charmhub with 1 image, deployed the charm, then uploaded/released a new image, and refreshed the charm. The image was updated successfully.

Changed in juju:
status: In Progress → Fix Committed
Changed in juju:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.