Offer permissions are not migrated

I have provided sosreports through internal channels.

The logs show the migration is failing because the model import to the target controller exits with an error. See the target model logs:

2022-01-12 23:51:39 ERROR juju.worker.migrationmaster.a5c742 worker.go:295 model data transfer failed, failed to import model into target controller: applications: application 0: application offer 0: application offer v2 schema check failed: acl: expected map, got nothing

Any exported application offers are required to have an "acl" attribute which is a map of who can access the offer and their permission. If that's missing, the import validation fails.

TL;DR; a failed migration, resulting in the operation being aborted and the model restored on the source controller appears to mess up offer permissions, resulting in any future migration attempts to fail (more detail below).

TL;DR; the original cause of the very first migration failure could have been anything, like a busy agent not quiesing, or a unit hook error etc; the point is the system gets wedged if that happens.

TL;DR; the reason the migration back after a failure messes up is that the offer permissions do not appear to get migrated correctly in the first place

Here's an snippet of a valid model export with an offer:

$ export JUJU_DEV_FEATURE_FLAGS=developer-mode
$ juju dump-model
...
applications:
  applications:
    charm-url: ch:amd64/trusty/mariadb-7
    name: mariadb
    offers:
      offers:
      - acl:
          admin: admin
          everyone@external: read
      - application-description: |
          MariaDB is an open source database server.
        application-name: mariadb
        endpoints:
          db: db
        offer-name: mariadb
        offer-uuid: d6ba3c46-4b01-4f10-838f-74c678ada51a
      version: 2
...

You can see that there are 2 grants that are done out of the box.

I migrated a test model with an offer and noticed some weird behaviour.
After the migration, doing a dump-model shows that there is no longer any "acl" block in the yaml.

And looking at the permissions collection shows that the offer permissions are indeed missing.

The result of the above is that attempting to migrate the model again will result in the observed error, as can be seen from this status snippet

$ juju status
Model Controller Cloud/Region Version SLA Timestamp Notes
foo test aws/ap-southeast-2 2.9.24.1 unsupported 12:43:59+10:00 migrating: aborted, removing model from target controller: model data transfer failed, failed to import model into target controller: applications: application 0: application offer 0: application offer v2 schema check failed: acl: expected map, got nothing

So there's definitely a bug to fix here in migrating offers.

I've renamed the bug to reflect the core issue which is causing the migration to fail, as reflected in the logs.

Another issue is that when a model is migrated, the offer permissions are not removed from the source model.

https://github.com/juju/juju/pull/13620

Excellent to see this is repaired in newer versions.

I've hit another cloud where it seems I'm having this exact issue again. Other engineers have attempted migrations, hit failures, etc. And it may have been on a pre-2.9.25 version of Juju. We're now on 2.9.42, but we have a model which is lacking the ACL fields.

Appropriate snippet of juju dump-model:

```
    name: grafana
    offers:
      offers:
      - application-description: |
          Grafana is the leading graph and dashboard builder for visualizing
          time series metrics.
        application-name: grafana
        endpoints:
          dashboards: dashboards
        offer-name: grafana
        offer-uuid: <REDACTED>
      version: 2
```

All other offers in this model have an acl entry; just this one does not. And I'm hitting the same error on attempting to migrate this model to a new controller: "model data transfer failed, failed to import model into target controller: applications: application 3: application offer 0: application offer v2 schema check failed: acl: expected map, got nothing"

Is there a suggested workaround for environments where this issue has already occurred? Should the CMR be broken/removed and the offer recreated from scratch, for example?

Based upon discussion with jameinel and testing on the environment in question, it looks like the ACL can be manually restored with the juju grant command, e.g.:

juju grant admin admin admin/lma.grafana
juju grant everyone@external read admin/lma.grafana

After that change, the ACL does show up in "juju dump-model", and the migration does seem to get further. I'm still hitting migration issues here, but I'm going to consider it a different issue at this point; I'll leave the above comment for future travelers in case it helps.