Simplestreams for rackspace are now live!

Hi Jose.

Juju cannot bootstrap. No images are found. Juju reads
    https://cloud-images.ubuntu.com/releases/streams/v1/index.sjson
and rackspace is not listed. Juju wont believe the data is like until the product file is in the index.

@Curtis

I am not convinced that this is the problem with CPC images.
When we bootstrapped and tested rackspace in CI, we have always provided a path to custom images. I am not convinced that we have ever filtered them by cloud name, for example, ec2 with "aws".

I am guessing that you have bootstrapped manually. I am not convinced that we are looking for "rax" image metadata. We filter simplestreams metadata by region, so it's important for Juju to be able to determine what region you are trying to bootstrap in.

Please add logs, preferably at TRACE level, to the bug as well as the commands and config that you have used.

It looks to me that index file contains rackspace images metadata under "rax" which, at a glance, is similar to what we have in CI for rackspace.

The only differences I am seeing:
* in index file:
- CI has cloudname: "released"; CPC Supplied metadata cloudname: "rax" [CPC's one looks correct to me];
- CI index file is not sjson; CPC's data is signed as it's in signed data.

I have not compared CPC products file with the one CI uses.

@anastaia, you are confused. These *new* stream are not CI's streams. As an author of streams. I am very aware about setting custom streams for tests. For the test of the proposed streams, I had to write and publish the streams myself. The streams you investigated last week are not related to these streams. CI is not involved.

Since my last comment, index.sjson has been update to include racsapace, but juju still does not recognise the clouds because the endpoints used by Juju (also published here)
    http://streams.canonical.com/juju/public-clouds.syaml
are not the endpoints in the CPC streams
    https://cloud-images.ubuntu.com/releases/streams/v1/index.sjson

Juju, like other simplestream apps matches the cloud by endpoint. CPC uses this form
    https://syd.images.api.rackspacecloud.com/v2
Juju uses
    https://identity.api.rackspacecloud.com/v2.0

You can verify this issue with an RC1 candidate.
    juju --debug bootstrap rax-cpc rackspace/syd

03:52:53 DEBUG juju.environs.bootstrap bootstrap.go:569 found 0 image metadata in default cloud images
03:52:54 DEBUG juju.environs.simplestreams simplestreams.go:454 skipping index "http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson" because of missing information: index file has no data for cloud {SYD https://identity.api.rackspacecloud.com/v2.0} not found
03:52:54 DEBUG juju.environs.bootstrap bootstrap.go:565 ignoring image metadata in default ubuntu cloud images: index file has no data for cloud {SYD https://identity.api.rackspacecloud.com/v2.0} not found
03:52:54 DEBUG juju.environs.bootstrap bootstrap.go:573 found 0 image metadata from all image data sources
...
03:52:57 ERROR cmd supercommand.go:458 failed to bootstrap model: no image metadata found

You can be very clear about which streams you want juju to check by setting image-metadata-url to the official streams and switching providers to observe how juju reads the clouds

    juju --debug bootstrap rax-cpc rackspace/syd --config image-metadata-url=http://cloud-images.ubuntu.com/releases
vs.
    juju --debug bootstrap gce-cpc google/us-central1 --config image-metadata-url=http://cloud-images.ubuntu.com/releases

^ rackspace 0, google 6.

The two options forward are for CPC to change endpoints at cloud-images, or for juju to update its own file
    https://github.com/juju/juju/blob/master/cloud/fallback-public-cloud.yaml

^ I favour juju switching to what the CPC/cloud wants to use.
    https://<region>.images.api.rackspacecloud.com/v2

@Curtis

Understood, although not confused:
I was comparing rax data from https://cloud-images.ubuntu.com/releases/streams/v1/index.sjson and related product file with the files that CI uses for rackspace tests specified with image-metadata-url in cloud-city.

I'll leave to you to decide how to proceed.

I think there is some confusion around rackspace URLs.

The way the API works is you first query the indentity API URL:

https://identity.api.rackspacecloud.com/v2.0

And then you access the rackspace service of choice on a particular region or datacenter. For instance for "images" on "syd" that would be:

https://syd.images.api.rackspacecloud.com/v2
 -> https:\\{region_code}.{service}.api.rackspace.com/v2

Those URLs are not hardcoded or using conventions specified by us. Instead CPC code uses the pyrax library:
https://github.com/rackspace/pyrax

You can see the full code here:
https://git.launchpad.net/~cloudware/cloudware/+git/cloud-image-manager/tree/cloud_image_manager/clouds/rax.py#n184

Endpoint URLs in line 195 are extracted from pyrax's library pyrax.identity.services['image'].{client}.{management_url}. See lines 184 & 187

I really believe this is the right source for that data.

You can use the following script to identify against rackspace cloud (using proper credentials) and it will dump in the response all the URLs for all services, including "images":

$ cat rax-auth.sh
curl -s -X POST https://identity.api.rackspacecloud.com/v2.0/tokens \
    -H "Content-Type: application/json" \
        -d '{
                "auth": {
                    "RAX-KSKEY:apiKeyCredentials": {
                 "username": "'$RAX_USERNAME'",
          "apiKey": "'$RAX_APIKEY'"
             }
         }
            }' | python -m json.tool

@Jose

I don't doubt that image streams are correct or how rackspace works, but juju is an idiot. It is not doing that you expect it to. The Juju rackspace-provider is a subclass of the juju openstack-provider that disable or changes several behaviours.

Juju is looking at
   https://cloud-images.ubuntu.com/releases/streams/v1/index.sjson
for
   {IAD https://identity.api.rackspacecloud.com/v2.0}

It does not know that
    {
     "region": "IAD",
     "endpoint": "https://iad.images.api.rackspacecloud.com/v2"
    },
is the equivalent, nor did it ask at https://identity.api.rackspacecloud.com/v2.0 for the region's endpoint.

The rax product file is good. There are difference between the official index.json and the test index.json is just the endpoints.
   http://5d117a7efe5779e2018e-77a1198a3891c2a6e76d39896aa23783.r47.cf1.rackcdn.com/images/streams/v1/index.json

I cannot work around this issue in streams or in Juju's public-clouds.yaml. Juju needs to fix the rackspace-provider or maybe CPC can find a way too add '"endpoint": "https://identity.api.rackspacecloud.com/v2.0"' to the index at cloud-images.

Just to be clear to CPC and Juju engineers. If the official streams included these definitions in "clouds": [], all juju betas and rcs would work without special config.

                {
                    "region": "IAD",
                    "endpoint": "https://identity.api.rackspacecloud.com/v2.0"
                },
                {
                    "region": "DFW",
                    "endpoint": "https://identity.api.rackspacecloud.com/v2.0"
                },
                {
                    "region": "HKG",
                    "endpoint": "https://identity.api.rackspacecloud.com/v2.0"
                },
                {
                    "region": "SYD",
                    "endpoint": "https://identity.api.rackspacecloud.com/v2.0"
                }

@sinzui, what is the special config needed to get this to work?

Thanks Curtis, I get the whole problem now.

While we (CPC) could hack the simplestreams field to have:

                {
                    "region": "{region}",
                    "endpoint": "https://identity.api.rackspacecloud.com/v2.0"
                },

Instead of:
    {
     "region": "{region}",
     "endpoint": "https://{region}.images.api.rackspacecloud.com/v2"
    },

I do believe juju should fix its code to know that:

https://identity.api.rackspacecloud.com/v2.0

Is a global URL expected to be accessed first before going to any regionalized service.

Please let us know if juju will fix this or we need to hack the urls.

Talked to Nate and we agreed the best way is to replicate the way things are done for other openstack clouds like hpcloud:
https://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:hpcloud.json

I will work on this tomorrow and try to get something for Nate to test by the morning East Coast time.

I have code that generates the simplestreams attached

MP is pending, but it would be good if someone could verify juju works with this one

Nate et al,

New streams with the change from images to identity endpoint for regions have been published @ http://cloud-images.ubuntu.com/releases/streams/v1/ Please test and verify these. Thanks.

Jose,

Please take a look at https://cloud-images-jenkins.canonical.com/job/OXYGEN-Generate_data to see how it signs the index.json to create index.sjson. The index.sjson file was not updated when CloudImages_RAX_Simplestreams was run, as such it was still showing the wrong region endpoints. As a work-around I ran OXYGEN_Generate_data to have it update index.sjson. RAX simplestreams generation should do this on its own. Thanks

https://code.launchpad.net/~josvaz/cloudware/+git/cloud-image-manager/+merge/306918
has been merged now

https://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:rax.json
is now updated

 "format": "products:1.0",
 "_aliases": {
  "crsn": {
   "HKG": {
    "region": "HKG",
    "endpoint": "https://identity.api.rackspacecloud.com/v2.0/"
   },
   "IAD": {
    "region": "IAD",
    "endpoint": "https://identity.api.rackspacecloud.com/v2.0/"
   },
   "DFW": {
    "region": "DFW",
    "endpoint": "https://identity.api.rackspacecloud.com/v2.0/"
   },
   "ORD": {
    "region": "ORD",
    "endpoint": "https://identity.api.rackspacecloud.com/v2.0/"
   },
   "SYD": {
    "region": "SYD",
    "endpoint": "https://identity.api.rackspacecloud.com/v2.0/"
   }

Would juju test these now?