| 2017-02-02 21:06:24 |
Francis Ginther |
bug |
|
|
added bug |
| 2017-02-02 21:06:24 |
Francis Ginther |
attachment added |
|
no-image-metadata-found.tar.gz https://bugs.launchpad.net/bugs/1661392/+attachment/4812225/+files/no-image-metadata-found.tar.gz |
|
| 2017-02-02 21:07:30 |
Francis Ginther |
information type |
Proprietary |
Public |
|
| 2017-02-02 21:07:48 |
Francis Ginther |
bug task added |
|
autopilot-log-analyser |
|
| 2017-02-02 21:07:54 |
Francis Ginther |
autopilot-log-analyser: assignee |
|
Francis Ginther (fginther) |
|
| 2017-02-02 21:07:57 |
Francis Ginther |
autopilot-log-analyser: status |
New |
In Progress |
|
| 2017-02-03 22:33:31 |
Francis Ginther |
autopilot-log-analyser: status |
In Progress |
Fix Released |
|
| 2017-02-03 22:33:34 |
Francis Ginther |
autopilot-log-analyser: status |
Fix Released |
Fix Committed |
|
| 2017-02-03 22:33:40 |
Francis Ginther |
autopilot-log-analyser: importance |
Undecided |
High |
|
| 2017-02-09 16:53:43 |
Francis Ginther |
description |
Found this in our CI:
https://ci.lscape.net/job/landscape-system-tests/5178
https://ci.lscape.net/job/landscape-system-tests/5174
The symptom is that the test_juju_bootstrap_kvm test which attempts a juju bootstrap on the autopilot cloud is failing with:
INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 16:57:59 INFO cmd cmd.go:141 Loading image metadata
INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 16:57:59 DEBUG juju.environs imagemetadata.go:112 obtained image datasource "keystone catalog"
INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 16:57:59 DEBUG juju.environs imagemetadata.go:112 obtained image datasource "default cloud images"
INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 16:57:59 DEBUG juju.environs imagemetadata.go:112 obtained image datasource "default ubuntu cloud images"
INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 16:57:59 DEBUG juju.environs.bootstrap bootstrap.go:575 constraints for image metadata lookup &{{{region1 http://10.96.54.253:5000/v2.0} [win2008r2 win2012hvr2 win2016nano genericlinux win2012r2 win2016 win81 precise raring saucy trusty yakkety win2012 quantal vivid wily xenial win8 win10 centos7 utopic zesty win2012hv win7] [amd64 i386 armhf arm64 ppc64el s390x] released}}
INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 16:57:59 DEBUG juju.environs.simplestreams simplestreams.go:454 skipping index "http://10.96.54.252:80/swift/v1/simplestreams/data/streams/v1/index.json" because of missing information: index file has no data for cloud {region1 http://10.96.54.253:5000/v2.0} not found
INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 16:57:59 DEBUG juju.environs.bootstrap bootstrap.go:583 ignoring image metadata in keystone catalog: index file has no data for cloud {region1 http://10.96.54.253:5000/v2.0} not found
INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 16:58:00 DEBUG juju.environs.bootstrap bootstrap.go:587 found 0 image metadata in default cloud images
INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 16:58:01 DEBUG juju.environs.simplestreams simplestreams.go:454 skipping index "http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson" because of missing information: index file has no data for cloud {region1 http://10.96.54.253:5000/v2.0} not found
INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 16:58:01 DEBUG juju.environs.bootstrap bootstrap.go:583 ignoring image metadata in default ubuntu cloud images: index file has no data for cloud {region1 http://10.96.54.253:5000/v2.0} not found
INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 16:58:01 DEBUG juju.environs.bootstrap bootstrap.go:591 found 0 image metadata from all image data sources
INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 16:58:01 INFO juju.provider.common destroy.go:20 destroying model "controller"
INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 16:58:01 INFO juju.provider.common destroy.go:31 destroying instances
INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 16:58:05 DEBUG juju.provider.openstack provider.go:1127 terminating instances []
INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 16:58:05 DEBUG goose <autogenerated>:22 performing API version discovery for "http://10.96.55.3:9696/"
INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 16:58:05 WARN goose <autogenerated>:38 discovered API versions: [{Version:{major:2 minor:0} Links:[{Href:http://10.96.55.3:9696/v2.0 Rel:self}] Status:CURRENT}]
INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 16:58:05 INFO juju.provider.common destroy.go:51 destroying storage
INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 16:58:05 DEBUG juju.provider.openstack cinder.go:81 volume URL: http://10.96.55.7:8776/v2/9584f0d124494d2db439687e461dc2c2
INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 16:58:09 ERROR cmd supercommand.go:458 failed to bootstrap model: no image metadata found
INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 16:58:09 DEBUG cmd supercommand.go:459 (error details: [{github.com/juju/juju/cmd/juju/commands/bootstrap.go:565: failed to bootstrap model} {github.com/juju/juju/environs/bootstrap/bootstrap.go:244: } {github.com/juju/juju/environs/bootstrap/bootstrap.go:593: no image metadata found}])
or simply "no image metadata found".
(a nicer pastebin is here: https://pastebin.canonical.com/178052/)
I took a closer look at 5178 and it appears that glance-simplestreams-sync uploaded the "16.04 LTS" image to glance at 16:09:27. Logs from this run are attached.
Will try to catch this on a live system. |
This problem has been observed several times during landscape autopilot CI testing. The observed failure is a failed juju bootstrap with the debug messages:
keystone charm: cs:~openstack-charmers-next/xenial/keystone-281
glance-simplestreams-sync charm: cs:xenial/glance-simplestreams-sync-11
[from console logs (copied to https://pastebin.canonical.com/178802/)]
05:40:32 INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 05:40:04 DEBUG juju.environs.bootstrap bootstrap.go:571 constraints for image metadata lookup &{{{region1 http://10.96.64.28:5000/v2.0} [xenial win2012hvr2 win2012r2 precise quantal raring utopic vivid win2008r2 win2012hv genericlinux zesty win2016 centos7 saucy trusty wily yakkety win8 win81 win10 win2012 win2016hv win2016nano win7] [amd64 i386 armhf arm64 ppc64el s390x] released}}
05:40:32 INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 05:40:05 DEBUG juju.environs.simplestreams simplestreams.go:454 skipping index "http://10.96.64.31:8080/v1/AUTH_5720f9bbc9d34ef59f1aa378be68463a/simplestreams/data/streams/v1/index.json" because of missing information: index file has no data for cloud {region1 http://10.96.64.28:5000/v2.0} not found
05:40:32 INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 05:40:05 DEBUG juju.environs.bootstrap bootstrap.go:579 ignoring image metadata in keystone catalog: index file has no data for cloud {region1 http://10.96.64.28:5000/v2.0} not found
05:40:33 INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 05:40:06 DEBUG juju.environs.bootstrap bootstrap.go:583 found 0 image metadata in default cloud images
05:40:33 INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 05:40:06 DEBUG juju.environs.simplestreams simplestreams.go:454 skipping index "http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson" because of missing information: index file has no data for cloud {region1 http://10.96.64.28:5000/v2.0} not found
05:40:33 INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 05:40:06 DEBUG juju.environs.bootstrap bootstrap.go:579 ignoring image metadata in default ubuntu cloud images: index file has no data for cloud {region1 http://10.96.64.28:5000/v2.0} not found
05:40:33 INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 05:40:06 DEBUG juju.environs.bootstrap bootstrap.go:587 found 0 image metadata from all image data sources
05:40:33 INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 05:40:06 INFO juju.provider.common destroy.go:20 destroying model "controller"
05:40:33 INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 05:40:06 INFO juju.provider.common destroy.go:31 destroying instances
05:40:37 INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 05:40:10 DEBUG juju.provider.openstack provider.go:1127 terminating instances []
05:40:37 INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 05:40:10 DEBUG goose <autogenerated>:22 performing API version discovery for "http://10.96.63.239:9696/"
05:40:37 INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 05:40:10 WARN goose <autogenerated>:38 discovered API versions: [{Version:{major:2 minor:0} Links:[{Href:http://10.96.63.239:9696/v2.0 Rel:self}] Status:CURRENT}]
05:40:37 INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 05:40:10 INFO juju.provider.common destroy.go:51 destroying storage
05:40:37 INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 05:40:10 DEBUG juju.provider.openstack cinder.go:81 volume URL: http://10.96.64.32:8776/v2/8a0556c51d51437ba1c1a3ae36e305fb
05:40:40 INFO ui.openstack.test_region_juju_bootstrap.RegionBootstrapTest.test_juju_bootstrap_kvm . 05:40:13 ERROR cmd supercommand.go:458 failed to bootstrap model: no image metadata found
Of note is the line: "index file has no data for cloud {region1 http://10.96.64.28:5000/v2.0} not found".
The simplesteams data it is parsing is:
[from http://10.96.64.31:8080/v1/AUTH_5720f9bbc9d34ef59f1aa378be68463a/simplestreams/data/streams/v1/index.json]
{
"format": "index:1.0",
"index": {
"auto.sync": {
"cloudname": "glance-simplestreams-sync-openstack",
"clouds": [
{
"endpoint": "http://10.96.64.57:5000/v2.0",
"region": "region1"
}
],
"datatype": "image-ids",
"format": "products:1.0",
"path": "streams/v1/auto.sync.json",
"products": [
"com.ubuntu.cloud:server:14.04:amd64",
"com.ubuntu.cloud:server:16.04:amd64"
],
"updated": "Thu, 09 Feb 2017 05:09:38 +0000"
}
},
"updated": "Thu, 09 Feb 2017 05:08:26 +0000"
}
There is a mismatch between the endpoint that juju is looking for, "http://10.96.64.28:5000/v2.0" and the one advertised in the index.json, "http://10.96.64.57:5000/v2.0". "10.96.64.28" is the VIP for keystone, while "10.96.64.57" is the private IP of keystone-0. Somehow, glance-simplestreams-sync was given the wrong IP to use for the endpoint and it populated that into the simplestreams data. |
|
| 2017-02-09 16:55:26 |
Francis Ginther |
attachment added |
|
lp-1661392-endpoint-mismatch.tar.gz https://bugs.launchpad.net/landscape/+bug/1661392/+attachment/4815948/+files/lp-1661392-endpoint-mismatch.tar.gz |
|
| 2017-02-09 16:57:42 |
Francis Ginther |
attachment added |
|
lp-1661392-juju-status.txt https://bugs.launchpad.net/landscape/+bug/1661392/+attachment/4815949/+files/lp-1661392-juju-status.txt |
|
| 2017-02-09 16:58:08 |
Francis Ginther |
attachment added |
|
lp-1661392-keystone.config https://bugs.launchpad.net/landscape/+bug/1661392/+attachment/4815950/+files/lp-1661392-keystone.config |
|
| 2017-02-09 16:58:33 |
Francis Ginther |
attachment added |
|
lp-1661392-glance-simplestreams-sync.config https://bugs.launchpad.net/landscape/+bug/1661392/+attachment/4815951/+files/lp-1661392-glance-simplestreams-sync.config |
|
| 2017-02-09 17:00:53 |
Francis Ginther |
bug task added |
|
glance-simplestreams-sync-charm |
|
| 2017-02-09 17:01:40 |
Francis Ginther |
summary |
"no image metadata found" during test_juju_bootstrap_kvm |
g-s-s uses the wrong IP for the keystone endpoint, leads to juju bootstrap failure |
|
| 2017-02-09 17:06:00 |
Francis Ginther |
bug task added |
|
keystone (Juju Charms Collection) |
|
| 2017-02-09 17:26:46 |
David Ames |
keystone (Juju Charms Collection): status |
New |
Triaged |
|
| 2017-02-09 17:26:58 |
David Ames |
keystone (Juju Charms Collection): importance |
Undecided |
High |
|
| 2017-02-10 20:55:22 |
Chad Smith |
landscape: milestone |
17.01 |
17.02 |
|
| 2017-02-14 22:14:01 |
David Britton |
landscape: status |
New |
Triaged |
|
| 2017-02-14 22:14:06 |
David Britton |
landscape: importance |
Undecided |
High |
|
| 2017-02-16 13:29:25 |
Sandor Zeestraten |
bug |
|
|
added subscriber Sandor Zeestraten |
| 2017-02-23 18:52:48 |
James Page |
charm-keystone: importance |
Undecided |
High |
|
| 2017-02-23 18:52:48 |
James Page |
charm-keystone: status |
New |
Triaged |
|
| 2017-02-23 18:52:51 |
James Page |
keystone (Juju Charms Collection): status |
Triaged |
Invalid |
|
| 2017-02-24 16:07:24 |
David Britton |
keystone (Juju Charms Collection): status |
Invalid |
Fix Released |
|
| 2017-03-16 15:00:41 |
Chad Smith |
landscape: status |
Triaged |
Fix Committed |
|
| 2017-04-06 14:55:10 |
Andreas Hasenack |
landscape: status |
Fix Committed |
Fix Released |
|
| 2019-05-30 11:48:08 |
Chris MacNaughton |
glance-simplestreams-sync-charm: status |
New |
Incomplete |
|
| 2020-01-30 15:33:33 |
Chris MacNaughton |
charm-keystone: status |
Triaged |
Fix Released |
|
| 2020-01-30 15:33:38 |
Chris MacNaughton |
glance-simplestreams-sync-charm: status |
Incomplete |
Invalid |
|
