cannot assign unit "jenkins/0" to new machine: invalid constraint value: arch=amd64 valid values are: [arm64 armhf]

For non amd64 workloads you generally have to specify an arch constraint. "juju deploy myapp --constraints arch=arm64"

This is required because with Charmhub there are different charm packages to download based on whether it is amd64 or arm64, and rather than trying to guess we just mandate that an architecture be set, but we might as well have a default for the common amd64 case.

it does work if I run `juju deploy jenkins --constraints="arch=arm64"`, but my point is it did work in the past without adding the architecture constraint (juju version 2.8.11), and also, couldn't juju recognize it is in a arm64 (as it worked for bootstrapping) and overwriting the architecture?

This was a known breaking change between 2.8.x and 2.9.x.

I think we can be a bit more clever with this, maybe consider the controller as well, but in reality, this is known and documented clearly when 2.9.x was released.

See "Architecture and series constraint selection for charms" from https://discourse.charmhub.io/t/juju-2-9-0-release-notes/4525 and "Architecture" from https://discourse.charmhub.io/t/charmhub/3991 (there are more documentation that I've added).

So I believe we've let people know here.

Regardless if this is a known limitation, I believe it is still a valid concern. With the rise of arm in the server space and dominance in edge/IoT, I think we should aim to fix this behavior where possible and remove the amd64 guessing (deferring to the user to set a model default for example).

I think Juju can do better here. Juju knows the arch of the machine on which the charm is to be deployed. It knows because either a placement directive is used to assign the charm to a pre-existing machine, or Juju will spin up a new machine with either a user specified or a default arch. Yes, charmhub now needs Juju to ask for a charm with the arch specified, but Juju should be able to figure out what that is. And if now, it should fail early and not wait till later to surface the error in status.
Yes, I know that the deploy workflow has Juju downloading the charm early, and only later assigning the charm to a machine, but that's not the user's problem to deal with.

To clarify a bit more, consider

Case 1

$ juju add-machine --constraints "arch=arm64"
Machine 2 added.
$ juju deploy foo --to 2

Juju knows that it is an arm64 machine so it should ask charmhub for an arm64 charm and fail early if one doesn't exist.

Case 2

$ juju set-model-constraints "arch=arm64"
$ juju deploy foo

Juju will use the model constraints set by the user to spin up arm64 machines whenever a new machine is needed, so again, Juju knows what arch should be used.

Case 3

$ juju deploy foo

Here, Juju will spin up a machine but doesn't know ahead of time what the arch will be. But once the arch is known, Juju can then ask for the relevant charm. The fact that Juju defaults up front to "amd64" and doesn't take account of the ultimate target machine arch is a bug IMO. Just because charms used to be arch agnostic and now they are now is not the user's problem to solve, when Juju does ultimately have all the info needed to make the required decision.

Looks like Harry and I were both editing the bug at the same time - I didn't see his comment before posting mine but we are both in broad agreement it seems :-)

Regarding Case #3 this could affect the possible introduction of default constraints into charm metadata/manifest. Possibly this means we either force a charm to specify default constraints on a per channel basis or defer to the user when default constraints are ambiguous.

This is related to bug 1969402 which talks about using model constraints as a partial workaround.
At this stage, any fix will not be in 2.9 so I'll remove the milestone.