stable juju cannot bootstrap because it selects the devel stanza in index.json

Though alpha3 is public and available on the CPCs, the release is blocked since we cannot provide alpha3 to private clouds.

This is record of the first report. We can see com.ubuntu.juju:released:tools is used to read the mirrors file. but the reading of the index.sjson is clearly using the devel stanza.
    https://pastebin.canonical.com/119982/

Note that the mirror files have a fixes order. released is always first, and the mirror files still support devel and even support future proposed.

I setup the json and 1.20.11-trusty-amd64 at
    http://people.canonical.com/~hduran/juju-dist/tools/streams/v1/index.json
and we can see buried in the details that devel was selected
    http://pastebin.ubuntu.com/8841954/
and juju fell back to streams.canonical.com.

The final line of the bootsrap confirms this
    Fetching tools: curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 -o $bin/tools.tar.gz 'https://streams.canonical.com/juju/tools/releases/juju-1.20.11-trusty-amd64.tgz'

Order is important! I moved com.ubuntu.juju:released:tools to the first position in index.json, then tried the bootrap again.
I can see from the --debug ouput that com.ubuntu.juju:released:tools.json was read, and this is confirmed by the final line.

Fetching tools: curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 -o $bin/tools.tar.gz 'http://people.canonical.com/~curtis/juju-dist/tools/releases/juju-1.20.11-trusty-amd64.tgz'

1. I propose the fix for this bug is to ensure the com.ubuntu.juju:released:tools stanza is always the first in indexjson.
2. Juju-qa will update assemble-streams to check for this case and abort when released is not first.
3. Juju-qa could rewrite the file to ensure the order is correct, allowing us to continue with the release with waiting for beta1

Oh, maybe this entire implementation is doomed. the listing in debugs sometime show maps. Since ppc64el uses gccgo, its 1.18.x and 1.20.x jujus will always select a stanza at random.

I added juju-release-tools because it needs to validate index.json's order, and it might be able to rewrite to complete the alpha3 release.

Will changing the order will work for most 1.18 and 1.20 archs, ppc64el uses gccgo which does not guarantee map order. Neither does golang 1.3 for that matter. We can see from this output that juju 1.20.10 ppc64el , selected devel even though released is first.
    http://pastebin.ubuntu.com/8842960/

The current thinking is that juju 1.21.x will create and look for index2.json

juju-release tools must prevent the deletion of index.json to ensure that historic juju finds an index.

> Will changing the order will work for most 1.18 and 1.20 archs, ppc64el uses gccgo which does not guarantee map order. Neither does golang 1.3 for that matter. We can see from this output that juju 1.20.10 ppc64el , selected devel even though released is first.

Please stop saying this, golang-go 1.20 that builds juju on intel also
provides no guarantees on map ordered, we've just been getting lucky.