Canonical Juju

Juju 3 - cannot refresh charms after revision tracking

Bug #2012700 reported by Matus Kosut on 2023-03-24

This bug affects 6 people

	Status	Importance	Assigned to	Milestone
Canonical Juju	Fix Released	High	Heather Lanigan	Canonical Juju 3.2-beta1
3.0	Fix Released	High	Heather Lanigan
3.1	Fix Released	High	Heather Lanigan	Canonical Juju 3.1.2

Bug Description

we have noticed strange issue with charm refresh on Juju 3+ deployment. Tested with multiple versions of juju controller (3+), e.g. 3.0.3 and 3.1.0.

There seems to be this new feature that tracks whether there is new charm revision available in the channel. Not sure how often this update is pulled (might take an hour or few for it to pop up). If there is a new revision found charm revision number gets highlighted in juju status.

The core of the issue is that once the highlight appears we are not able to run refresh anymore. Instead an error is returned:

```
ubuntu@juju-dev--multipass-juju:~$ juju refresh rsyslog-forwarder
ERROR already exists
```

To reproduce this one has to deploy older version of a charm and then wait for juju to fetch revisions and mark that the newer one exists.

See original description

Revision history for this message

Heather Lanigan (hmlanigan) wrote on 2023-03-24:

The charm revision pieces has been around since the beginning of juju 2.x. What has changed in juju 3 is async download of charm and surrounding logic.

I'm surprised we're only seeing this in one place as the charm revision data is gathered, and potentially updated every 24 hours.

Changed in juju:
assignee:	nobody → Heather Lanigan (hmlanigan)
status:	New → Triaged
importance:	Undecided → High
milestone:	none → 3.2-beta1

Matus Kosut (matuskosut) on 2023-03-26

description:

updated

Revision history for this message

Adam Dyess (addyess) wrote on 2023-03-28:

just to add another point to this bug, i tried with python juju==3.1.0.1

```python
    for app_name, app in model.applications.items():
        log.info(f"Upgrading {app_name} from {app.charm_url} to --channel={channel}")
        await app.refresh(channel=channel)
```

and found the same error is produced. I guess this likely indicates an issue on the controller

Revision history for this message

John A Meinel (jameinel) wrote on 2023-03-28:

I'm guessing that the work to update the tracker to notice that there are new resources available as well as a new charm revision, ends up putting something into the database such that the 'juju refresh' action then says "oh, I can't do that, you already have it". We'll have to figure out why that is an issue, but it very much seems to be a controller side issue.

Revision history for this message

Heather Lanigan (hmlanigan) wrote on 2023-03-28:

The work referred to in #3 has not been done yet. Investigating the cause now.

Changed in juju:
status:	Triaged → In Progress

Revision history for this message

Heather Lanigan (hmlanigan) wrote on 2023-03-28:

Must update the error as well, it is unhelpful: "ERROR already exists"

Revision history for this message

Heather Lanigan (hmlanigan) wrote on 2023-03-29:

This appears to be a day one bug in juju 3.0 caused by enabling asynchronous charm download for charmhub charms by default. In juju 2.9 it was behind a feature flag.

This issue is specific to charmhub charms only.

To download a charm asynchronously, juju inserts a charm doc with some the charm's data and indicated that it requires downloading. Naively assuming that if the charm doc already exists, that there is nothing to do, the charm has been downloaded. However if the charm revision updater worker has run and found a charm revision different that the current one, it inserts a charm doc into the juju db with a placeholder flag.

Revision history for this message

Heather Lanigan (hmlanigan) wrote on 2023-03-29:

https://github.com/juju/juju/pull/15382 for 3.0

Heather Lanigan (hmlanigan) on 2023-03-31

Changed in juju:
status:	In Progress → Fix Committed

Revision history for this message

Matus Kosut (matuskosut) wrote on 2023-04-20:

@hmlanigan I briefly tested the upgrades with newly released v3.1.2 and it seems to work. Although I noticed there is multiple failed retries before it actually downloads a new charm revision. I wasn't sure if that is expected behavior so I link the logs in case it might be useful:

https://gist.github.com/matuskosut/d01f1424ed066520c5c6b6bfcf2f7980