when launching a tar-root image (xenial cloud image) on a nova-compute lxc hypervisor i get
2017-05-12 09:36:35.270 9689 WARNING nova.scheduler.client.report [req-e3ce840a-b823-40ce-9c32-85b99e118a70 8446f64e0b504345b48673a3e3a328f1 35515180b8b646329e2caa2372250e0b - - -] Unable to refresh my resource provider record
2017-05-12 09:36:53.532 9689 ERROR nova.virt.lxd.driver [req-e3ce840a-b823-40ce-9c32-85b99e118a70 8446f64e0b504345b48673a3e3a328f1 35515180b8b646329e2caa2372250e0b - - -] [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] Failed to upload to LXD: Image is unacceptable: Bad Image format: Image could not be found.
2017-05-12 09:36:53.533 9689 ERROR nova.compute.manager [req-e3ce840a-b823-40ce-9c32-85b99e118a70 8446f64e0b504345b48673a3e3a328f1 35515180b8b646329e2caa2372250e0b - - -] [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] Instance failed to spawn
2017-05-12 09:36:53.533 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] Traceback (most recent call last):
2017-05-12 09:36:53.533 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2082, in _build_resources
2017-05-12 09:36:53.533 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] yield resources
2017-05-12 09:36:53.533 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1924, in _build_and_run_instance
2017-05-12 09:36:53.533 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] block_device_info=block_device_info)
2017-05-12 09:36:53.533 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] File "/usr/lib/python2.7/dist-packages/nova/virt/lxd/driver.py", line 274, in spawn
2017-05-12 09:36:53.533 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] self.setup_image(context, instance, image_meta)
2017-05-12 09:36:53.533 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] File "/usr/lib/python2.7/dist-packages/nova/virt/lxd/driver.py", line 1333, in setup_image
2017-05-12 09:36:53.533 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] os.unlink(container_manifest)
2017-05-12 09:36:53.533 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-05-12 09:36:53.533 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] self.force_reraise()
2017-05-12 09:36:53.533 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-05-12 09:36:53.533 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] six.reraise(self.type_, self.value, self.tb)
2017-05-12 09:36:53.533 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] File "/usr/lib/python2.7/dist-packages/nova/virt/lxd/driver.py", line 1216, in setup_image
2017-05-12 09:36:53.533 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] image_id=instance.image_ref, reason=reason)
2017-05-12 09:36:53.533 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] ImageUnacceptable: Image is unacceptable: Bad Image format: Image could not be found.
2017-05-12 09:36:53.533 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840]
2017-05-12 09:36:53.533 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840]
2017-05-12 09:36:53.549 9689 WARNING nova.virt.lxd.driver [req-e3ce840a-b823-40ce-9c32-85b99e118a70 8446f64e0b504345b48673a3e3a328f1 35515180b8b646329e2caa2372250e0b - - -] Failed to delete instance. Container does not exist for instance-00000097.
2017-05-12 09:36:54.085 9689 WARNING nova.compute.manager [req-e3ce840a-b823-40ce-9c32-85b99e118a70 8446f64e0b504345b48673a3e3a328f1 35515180b8b646329e2caa2372250e0b - - -] Could not clean up failed build, not rescheduling. Error: Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf zfs destroy lxd/instance-00000097-ephemeral
Exit code: 1
Stdout: u''
Stderr: u"cannot open 'lxd/instance-00000097-ephemeral': dataset does not exist\n"
2017-05-12 09:36:54.277 9689 WARNING nova.scheduler.client.report [req-e3ce840a-b823-40ce-9c32-85b99e118a70 8446f64e0b504345b48673a3e3a328f1 35515180b8b646329e2caa2372250e0b - - -] No authentication information found for placement API. Placement is optional in Newton, but required in Ocata. Please enable the placement service before upgrading.
2017-05-12 09:36:54.278 9689 WARNING nova.scheduler.client.report [req-e3ce840a-b823-40ce-9c32-85b99e118a70 8446f64e0b504345b48673a3e3a328f1 35515180b8b646329e2caa2372250e0b - - -] Unable to refresh my resource provider record
2017-05-12 09:36:54.280 9689 ERROR nova.compute.manager [req-e3ce840a-b823-40ce-9c32-85b99e118a70 8446f64e0b504345b48673a3e3a328f1 35515180b8b646329e2caa2372250e0b - - -] [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] Build of instance 1ad75f54-e9a3-420f-bd4c-defe890fa840 aborted: Image is unacceptable: Bad Image format: Image could not be found.
2017-05-12 09:36:54.280 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] Traceback (most recent call last):
2017-05-12 09:36:54.280 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1783, in _do_build_and_run_instance
2017-05-12 09:36:54.280 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] filter_properties)
2017-05-12 09:36:54.280 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1943, in _build_and_run_instance
2017-05-12 09:36:54.280 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] 'create.error', fault=e)
2017-05-12 09:36:54.280 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-05-12 09:36:54.280 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] self.force_reraise()
2017-05-12 09:36:54.280 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-05-12 09:36:54.280 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] six.reraise(self.type_, self.value, self.tb)
2017-05-12 09:36:54.280 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1927, in _build_and_run_instance
2017-05-12 09:36:54.280 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] instance=instance)
2017-05-12 09:36:54.280 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
2017-05-12 09:36:54.280 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] self.gen.throw(type, value, traceback)
2017-05-12 09:36:54.280 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2109, in _build_resources
2017-05-12 09:36:54.280 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] reason=six.text_type(exc))
2017-05-12 09:36:54.280 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840] BuildAbortException: Build of instance 1ad75f54-e9a3-420f-bd4c-defe890fa840 aborted: Image is unacceptable: Bad Image format: Image could not be found.
2017-05-12 09:36:54.280 9689 ERROR nova.compute.manager [instance: 1ad75f54-e9a3-420f-bd4c-defe890fa840]
2017-05-12 09:36:55.391 9689 WARNING nova.compute.manager [req-e3ce840a-b823-40ce-9c32-85b99e118a70 8446f64e0b504345b48673a3e3a328f1 35515180b8b646329e2caa2372250e0b - - -] Failed to delete volume: 7c49db87-16ab-4b30-9b13-832bd7cbf720 due to Invalid input received: Invalid volume: Volume status must be available or error or error_restoring or error_extending and must not be migrating, attached, belong to a group or have snapshots. (HTTP 400) (Request-ID: req-f82b164d-9044-4661-ab69-0bdbab083b4b)
it looks like nova cannot retrieve image from glance.
I checked /etc/nova/nova.conf in kvm nodes and lxd hypervisor nodes and they look the same.
Regarding glance section it just contains the same endpoint
[glance]
api_servers = http://10.10.10.95:9292
Following the backtrace it seems it cannot get the image metadata at all. I checked the upstream source at https://github.com/openstack/nova-lxd/blob/master/nova/virt/lxd/driver.py#L398
has changed the galnce-lxd sync code.
Maybe it has been fixed already, did anyone try? do we have a bleeding edge nova-lxd ppa to test against?
hi all, is anyone checking this issue?