Juju 3.1-rc2 fails to bootstrap LXD controller on arm64

To add here: It works fine to bootstrap a controller on 2.9 without the use of --bootstrap-constraints. This looks like a regression somewhere.

I try with using bootstrap --constraints='arch=amd64' But i have the same issue!!!

This looks like you've not properly configure juju to the proposed stream. For a rc, you need to set the agent-stream config to proposed, or else juju cannot find the packaged agent binaries.

See if it works now that 3.1.0 is out. For the next rc, you'd need to

juju bootstrap lxd --model-default agent-stream=proposed

The interesting thing is that it was working without problems when I added --bootstrap-constraints='arch=arm64', no need to set the agent-stream.

I just tried again to bootstrap with juju from 3.1/stable on a fresh arm64 machine and the problem is still the same:

$ sudo lxd init --auto
$ snap install --channel=3.1/stable juju
$ mkdir -p .local/share/juju
$ juju bootstrap lxd test
...
ERROR cannot deploy controller application: cannot add controller application: cannot add application "controller": invalid constraint value: arch=amd64
valid values are: [arm64 armhf]
ERROR failed to bootstrap model: subprocess encountered error code 1

Now running the following instead bootstraps the controller without problems:

$ juju bootstrap lxd test --bootstrap-constraints arch=arm64

With the transition to multi-arch charms, arch needs to be explicitly specified. It defaults to amd64. For most bootstraps, eg aws, maas, other clouds, or lxd on amd64, bootstrap does not need arch to be explicitly specified because the controller instance is amd64. But because you're on an arm64 host, the lxd instance is also arm64 but juju doesn't have that coded into the generic bootstrap logic; the default arch of amd64 is used. What juju could do is inspect the arch of the bootstrapped controller and wire that through as the controller application arch. But subsequent deploys to that model would still use amd64 unless we also set the model constraint arch as well. It is probably ok, but we'd need to think through the implications.

The reason this wasn't an issue in 2.9 is because 2.9 doesn't install a controller charm at bootstrap.

We can consider adding a special case for running the localhost provider on the host machine (not a lxd remote cluster). In this case, we would set the bootstrap arch to the host arch, and also model default constraint arch to host arch (if not amd64 and not explicitly set). If explicitly set to something incompatible, we would error early. The narrow scope of this special case avoids trying to be too clever in the general sense - we always prefer explicit over implicit behaviour.

We have fixed this on one side, the bootstrapping:

https://github.com/juju/juju/pull/15269

On a later PR (after we add the feature) we'll be able to add model-default constraints at bootstrap which will avoid having to pass the (arch) constraint to the newly created model.