[2.1.1] Controller does not look at agent_stream in model-config - it uses agent_stream that it booted with

The version number 2.1.1.1 suggests that the devel streams were not used for the original bootstrap. x.x.x.1 numbers are created when juju uploads the agent itself.

To avoid conflating issues, could you try to reproduce using the actual devel streams?

@abentley I have previously confirmed through test and speaking with Curtis and Nicholas that the devel streams don't have the arm agents.

The devel streams don't contain any agents for 2.1.1.

They do contain arm agents (among others) for devel releases:
$ sstream-query http://streams.canonical.com/juju/tools/streams/v1/index2.json content_id='com.ubuntu.juju:devel:tools' -o"%(item_name)s"|grep ^2.1.*arm |sort
2.1-beta1-trusty-arm64
2.1-beta1-xenial-arm64
2.1-beta1-yakkety-arm64
2.1-beta1-zesty-arm64
2.1-beta2-trusty-arm64
2.1-beta2-xenial-arm64
2.1-beta2-yakkety-arm64
2.1-beta2-zesty-arm64
2.1-beta3-trusty-arm64
2.1-beta3-xenial-arm64
2.1-beta3-yakkety-arm64
2.1-beta3-zesty-arm64
2.1-beta4-trusty-arm64
2.1-beta4-xenial-arm64
2.1-beta4-yakkety-arm64
2.1-beta4-zesty-arm64
2.1-beta5-precise-arm64
2.1-beta5-trusty-arm64
2.1-beta5-xenial-arm64
2.1-beta5-yakkety-arm64
2.1-beta5-zesty-arm64
2.1-rc1-precise-arm64
2.1-rc1-trusty-arm64
2.1-rc1-xenial-arm64
2.1-rc1-yakkety-arm64
2.1-rc1-zesty-arm64
2.1-rc2-precise-arm64
2.1-rc2-trusty-arm64
2.1-rc2-xenial-arm64
2.1-rc2-yakkety-arm64
2.1-rc2-zesty-arm64

@Aaron yes, 2.1.1 is released. The location of the agents is not what this bug covers. The bug is that the controller is not seeing or honoring the model setting for agent-stream. It appears to be permanently set to what it was when the controller was bootstrapped. I had to destroy the controller and re-bootstrap to get this to work.

This is interesting because it will put Juju controller in an unknown state - I do not believe we have encountered situations where models would be created with different agent streams.

You will end up with some models using devel agents, released agents and all other combinations in between.

I think we intentionally use agent stream from controller to ensure homogeneity. But I will bring it up with the team.

Just commenting that bug 1674755 is very similar. I wonder what other settings may be "fixed" at bootstrap.

Referring to the original bug description, when adding a model and deploying machines to that model, the running Juju agent version influences the stream used. If the running Juju version is a development version, (which it is in this case going by the version number) then I think the stream used for subsequent tools selection will be "develop", regardless of what any model agent stream is set to. Not saying this is correct but it explains the observed behaviour.

I've just noticed that I'm getting 'released' images for my model instances, despite setting image-stream to daily. The controller instances, however, do use a daily image as expected. This is Juju 2.1.2 + OpenStack Provider against an Ocata undercloud.

#### On a user's model, the instance is based on the *release* image, when the *daily* image was expected.

jenkins@juju-10f68a-osci-13:~$ openstack server show f487418d-0fe8-46ec-a8e1-92ff859c9051 | grep image
| image | ubuntu-released/ubuntu-xenial-16.04-amd64-server-20170330-disk1.img (8b4dc7a6-11ad-44a0-86ca-9844d9763704) |

jenkins@juju-10f68a-osci-13:~$ openstack server show 5d799d64-cb31-475a-ac78-59fff602e0f3 | grep image
| image | ubuntu-released/ubuntu-xenial-16.04-amd64-server-20170330-disk1.img (8b4dc7a6-11ad-44a0-86ca-9844d9763704)

jenkins@juju-10f68a-osci-13:~$ juju switch auto-osci-sv08
auto-osci-sv08:admin/controller (no change)

jenkins@juju-10f68a-osci-13:~$ juju model-config
Attribute From Value
agent-metadata-url default ""
agent-stream controller proposed
agent-version model 2.1.2
apt-ftp-proxy default ""
apt-http-proxy default ""
apt-https-proxy default ""
apt-mirror default ""
automatically-retry-hooks controller false
default-series controller xenial
development default false
disable-network-management default false
enable-os-refresh-update controller false
enable-os-upgrade controller false
external-network default ""
firewall-mode default instance
ftp-proxy default ""
http-proxy default ""
https-proxy default ""
ignore-machine-addresses default false
image-metadata-url default ""
image-stream controller daily
logforward-enabled default false
logging-config model <root>=WARNING;unit=DEBUG
net-bond-reconfigure-delay default 17
network model 86125222-dd54-4df2-9421-235456fab193
no-proxy default ""
provisioner-harvest-mode default destroyed
proxy-ssh default false
resource-tags model {}
ssl-hostname-verification default true
storage-default-block-source model cinder
test-mode controller true
transmit-vendor-metrics controller false
use-default-secgroup controller true
use-floating-ip default false

.

#### On the controller model, the instance is based on the *daily* image as expected.

jenkins@juju-10f68a-osci-13:~$ juju switch controller
auto-osci-sv08:admin/auto-osci-sv08 -> auto-osci-sv08:admin/controller

jenkins@juju-10f68a-osci-13:~$ juju machines
Machine State DNS Inst id Series AZ
0 started 172.17.108.3 e872d229-1f50-4d2a-9263-b8ade...

I tried again with agent-stream: released and image-stream: daily (with Juju 2.1.2 from the stable ppa) on a fresh controller and observed the same:

Attribute From Value
agent-metadata-url default ""
agent-stream controller released
agent-version model 2.1.2
apt-ftp-proxy default ""
apt-http-proxy default ""
apt-https-proxy default ""
apt-mirror default ""
automatically-retry-hooks controller false
default-series controller xenial
development default false
disable-network-management default false
enable-os-refresh-update controller false
enable-os-upgrade controller false
external-network default ""
firewall-mode default instance
ftp-proxy default ""
http-proxy default ""
https-proxy default ""
ignore-machine-addresses default false
image-metadata-url default ""
image-stream controller daily
logforward-enabled default false
logging-config model <root>=WARNING;unit=DEBUG
net-bond-reconfigure-delay default 17
network model 86125222-dd54-4df2-9421-235456fab193
no-proxy default ""
provisioner-harvest-mode default destroyed
proxy-ssh default false
resource-tags model {}
ssl-hostname-verification default true
storage-default-block-source model cinder
test-mode controller true
transmit-vendor-metrics controller false
use-default-secgroup controller true
use-floating-ip default false

Model Controller Cloud/Region Version
auto-osci-sv08 auto-osci-sv08 serverstack/serverstack 2.1.2

App Version Status Scale Charm Store Rev OS Notes
ubuntu waiting 0/4 ubuntu jujucharms 10 ubuntu

Unit Workload Agent Machine Public address Ports Message
ubuntu/0 waiting allocating 0 172.17.108.10 waiting for machine
ubuntu/1 waiting allocating 1 172.17.108.12 waiting for machine
ubuntu/2 waiting allocating 2 172.17.108.14 waiting for machine
ubuntu/3 waiting allocating 3 172.17.108.3 waiting for machine

Machine State DNS Inst id Series AZ
0 pending 172.17.108.10 a2e6d361-e72f-4b85-aa2d-6383ce3ab8a0 xenial nova
1 pending 172.17.108.12 9614d394-58f7-49a7-9578-1267cc7aa902 xenial nova
2 pending 172.17.108.14 cf6c3952-1f71-4e6b-991e-295ea5703931 xenial nova
3 pending 172.17.108.3 4072c647-2c7b-4eba-b085-3d507662cd24 xenial nova

(clients) jenkins@juju-10f68a-osci-13:~/tools/foo/charm-test-infra$ openstack server list | grep auto-osci-sv08
| 4072c647-2c7b-4eba-b085-3d507662cd24 | juju-902164-auto-osci-sv08-3 | ACTIVE | auto-osci_...

@Ryan, in our case, the workaround I used for changing agent_stream was to bootstrap a new controller with the right settings. That seemed to be the only way to make the switch.

I believe this is fixed now in 2.3-rc2, with the other upgrade changes.