Feature request: juju support for nested virtualisation images on GCP

For info; the GCP provider for juju doesn't seem to follow the standard pattern of simplestreams.

Instead, environ_broker.go creates the image path by appending ubuntuImageBasePath + spec.Image.Id

ubuntuImageBasePath is hardcoded in gce.go to projects/ubuntu-os-cloud/global/images/

Image.Id appears to come from simplestreams download but I can't see where just yet.

We would happily provide a local simplestreams json to point to our own custom GCP image, but because the ubuntuImageBasePath is hardcoded in juju, we can't tell juju to use any of our own images.

The available image ids for the GCE regions is queried at bootstrap time and is sourced from here:

http://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:gce.json

TL;DR; It's possible to copy and edit the metadata or generate your own and place it in a directory (you need the index file and the file above). You then bootstrap with --metadata-source=<dir>. More info in this post https://discourse.jujucharms.com/t/juju-deploy-series-centos/1294/3

If you did go and create your own images, I do not know if said images would still be available from the base path "projects/ubuntu-os-cloud/global/images/"? I suspect we'd also need to introduce a model config item to allow the base image path to be specified.

Hi Ian,

Thanks for the update.

>If you did go and create your own images, I do not know if said images would still be available from the base path "projects/ubuntu-os-cloud/global/images/"? I suspect we'd also need to introduce a model config item to allow the base image path to be specified.

That's the problem that I was hitting; that base path is hardcoded in the Juju so even if I overrode the image ids in metadata they would still be looking in the ubuntu GCP project.

Is it possible to introduce the suggested config item so that the image base path can be overridden please?

Thanks!

Ed - thanks for the report. Thanks for your patience while other issues were being addressed.

Code to enable this change is currently under review. Pending approval, we expect to have it available to you early next week via Juju's next beta release.

If you would like to track the progress in more detail, here is the direct link https://github.com/juju/juju/pull/10051

Thanks for the update - looking forward to the beta

FYI we have been testing this successfully, so thanks for the update.

Follow up related question - is it possible to update the image metadata after bootstrapping a controller with the GCE provider (without rebuilding the controller).

I've tried to set image-metadata-url in model-defaults to a URL that hosts the metadata index.json and com.ubuntu.cloud-released-imagemetadata.json files, however, I don't see the Juju controller trying to even hit this URL during provisioning of a new VM into the model.

@Ed Stewart (emcs2),

Did you create a new model after changing model-defaults?

model-defaults are the default config that new models get.

If you needed to change a config on an existing model, see 'juju help model-config'.

Also image-metadata-url is actually a controller-config. So on a bootstrap controller, you'd want to run 'juju controller-config image-metadata-url=<ur new url>', see 'juju help controller-config' for configuration keys. You might also ensure that image-stream is correct.

Thanks for the quick response

yep created a new model after model-defaults change and checked that the URL was set:

ubuntu@juju-jumphost:~$ juju model-config | grep image
base-image-path controller projects/myproject/global/images/
container-image-metadata-url default ""
container-image-stream default released
image-metadata-url controller https://mywebserver/repository/raw-hosted/gcp/simplestreams/metadata/images
image-stream default released

But it seems to be ignoring image-metadata-url and i see no GETs in mywebserver's logs from the Juju controller (i've obfuscated my URLs above, but mywebserver does include a valid trusted cert)

I also tried juju controller-config, but I get the following:

ubuntu@juju-jumphost:~$ juju controller-config image-metadata-url=https://mywebserver/repository/raw-hosted/gcp/simplestreams/metadata/images
ERROR unknown controller config setting "image-metadata-url"

Thanks

Hmm, I was playing around with 'controller-config' to answer you but was actually looking at the output of 'juju help model-config' to get config keys... :D You are absolutely right - image-metadata-url is a model property.
However, I am a bit 'surprised' that the new image-metadata-url seems to have no effect. I'll look into it my tomorrow. Any chance you could provide logs around the time when you would have expected the URL to be hit when provisioning a new VM into the model. I would have love to see a TRACE level but this might be very verbose...

Thanks!

Have uploaded logs to ubuntu support server with filename sosreport-juju-6ac7ce-0.launchpadbug1814633-20190716135645.tar.xz

These are from the controller after setting controller logging level to "<root>=TRACE;unit=TRACE"

@Ed Stewart (emcs2),

I am not sure how to access them on ubuntu support server. If these logs do not have any private info, they could be attached here.

And also, another question....

Juju caches images on the controller to minimise image downloads and, hence, traffic, to enable consistent deployment of several units. We cache them but we expire them after 5 mins. This means that if there are cached images that match your requirements, we will not re-download them even if you update image-metadata-url (maybe we should rect to that change in the future but, at the moment, we do not track where the images came from and to react to image-metadata-url changes, we'd need to do that).
So, the question is... Has 5 mins passed since the last time a deployment/machine provisioning happenned? I believe that if there are no cached images, we'd try the image-metadata-url... One way to know would be to change image-metadata-url and try deployment after 6+ mins... :D

I'll need to obfuscate the logs which may take some time. Some quick greps:

root@juju-6ac7ce-0:/var/log/juju# grep "provider.gce" machine-0.log
2019-07-16 17:01:50 DEBUG juju.provider.gce environ_broker.go:225 GCE user data; 4680 bytes
2019-07-16 17:01:50 INFO juju.provider.gce environ_broker.go:270 fetching disk image from projects/obfuscated/global/images/diskbionic-hv
2019-07-16 17:01:52 INFO juju.provider.gce.gceapi raw.go:383 GCE operation "operation-1563296510796-58dcf55346197-8cd18baa-b1159555", waiting...
2019-07-16 17:02:18 INFO juju.provider.gce.gceapi raw.go:404 GCE operation "operation-1563296510796-58dcf55346197-8cd18baa-b1159555" finished
2019-07-16 17:02:18 INFO juju.provider.gce environ_broker.go:58 started instance "juju-b95a7b-0" in zone "us-east1-b"

However, the 5 mins definitely passed (I redeployed an ubuntu charm onto a new vm just now with the same issue).

I have opened a separate report to deal with changed image-metadata-url (bug # 1836814) since the request reported here has been addressed. We should continue this conversation there. While opening a new report, I have discovered a bug that was describing similar failures around image-stream.