Config change fails in 2.7 due to UpdateNetworkInfo not being implemented

Joe, can you and Achillieas look into this in your AM and see if this is something we've broken at the moment in the path to 2.7?

This is weird because the controller has to be newer than the connecting agents and so such an error would never occur. However, during development, there could be a 2.7-beta1 and a different 2.7-beta1 which report the same version but one doesn't have the new API compiled in. I suspect that's what's happened here.

Ken, I think we wanted a sanity check on replicating this and I am going to mark this incomplete and see if it repeated or was due to upgrading things in a dev setup (2.7-beta1 overlap) as Ian was thinking.

I just tried deploying a Charmed Kubernetes cluster from scratch after doing a `snap refresh`, and I'm still getting this issue.

If it matters, I'm getting an error about `Uniter(12).UpdateNetworkInfo` not being implemented, and this comment says that only v13 is implemented:

https://github.com/juju/juju/blob/94dfc42/apiserver/facades/agent/uniter/uniter.go#L39-L40

The deployment scenario is:

bootstrap AWS controller
deploy CDK
add-k8s
deploy kubeflow

So the binary that is out of date is not the jujud operator as i thought but the normal cloud jujud.

Looking at what is deployed on an IAAS controller, the jujud is way out of date:

-rw-r--r-- 1 root root 225 Oct 16 23:22 downloaded-tools.txt
-rwxr-xr-x 1 root root 117408800 Oct 8 12:38 jujud
ubuntu@ip-10-218-201-209:~$ cat /var/lib/juju/tools/2.7-beta1-bionic-amd64/downloaded-tools.txt
{"version":"2.7-beta1-bionic-amd64","url":"https://juju-dist.s3.amazonaws.com/tools/agent/2.7-beta1/juju-2.7-beta1-ubuntu-amd64.tgz","sha256":"225dcd7348241fc3fb8ca17f17aeaf83b9bf3c99d5cd765e73c0165222df17a6","size":33295612}ubuntu@ip-10-218-201-209:~$

It is from October 8. So our packaged agent binaries are old.

I bootstrapped to GCE and confirmed that the correct version of jujud is deployed so it appears that it is just the AWS juju mirrors that are stale

Listing the content of the AWS bucket confirms it

tools/agent/2.7-beta1/juju-2.7-beta1-centos7-amd64.tgz 2019-10-08T06:28:30Z 46M
tools/agent/2.7-beta1/juju-2.7-beta1-ubuntu-amd64.tgz 2019-10-08T06:29:04Z 31M
tools/agent/2.7-beta1/juju-2.7-beta1-ubuntu-arm64.tgz 2019-10-08T06:29:18Z 29M
tools/agent/2.7-beta1/juju-2.7-beta1-ubuntu-ppc64el.tgz 2019-10-08T06:29:30Z 28M
tools/agent/2.7-beta1/juju-2.7-beta1-ubuntu-s390x.tgz 2019-10-08T06:29:39Z 30M
tools/agent/2.7-beta1/juju-2.7-beta1-windows-amd64.tgz 2019-10-08T06:29:48Z 48M

So I'm surprised we have those agent binaries published for an unreleased juju. This I think is what's breaking things. Normally:

the snap will look for published binaries and use those if found
if not, it will used the jujud in the snap

The fact that we appear to have published as a one off 2.7 beta1 binaries means that we will use those stale binaries by mistake in preference to the correct jujud.

Sorry for this. This was due to me testing the new juju release process last week for the new up and coming juju release. The changes originally should have been temporary, but ended up sticking in AWS and Azure.

I've cleaned up all remaining remnants of the "2.7beta1" test release from the stream json files and any lingering agent binaries built.

I'm currently detailing the process to correctly rollback a streams release for future reference.

Let me know if you're still having issues.