Tested on OSP12 (Pike), but appears to be still present on master. Should only occur if nova compute is configured to use local file instance storage.
Create instance A on compute X
Resize instance A to compute Y
Domain is powered off
/var/lib/nova/instances/<uuid A> renamed to <uuid A>_resize on X
Domain is *not* undefined
On compute X:
update_available_resource runs as a periodic task
First action is to update self
rt calls driver.get_available_resource()
...calls _get_disk_over_committed_size_total
...iterates over all defined domains, including the ones whose disks we renamed
...fails because a referenced disk no longer exists
Results in errors in nova-compute.log:
2018-05-30 02:17:08.647 1 ERROR nova.compute.manager [req-bd52371f-c6ec-4a83-9584-c00c5377acd8 - - - - -] Error updating resources for node compute-0.localdomain.: DiskNotFound: No disk at /var/lib/nova/instances/f3ed9015-3984-43f4-b4a5-c2898052b47d/disk
2018-05-30 02:17:08.647 1 ERROR nova.compute.manager Traceback (most recent call last):
2018-05-30 02:17:08.647 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6695, in update_available_resource_for_node
2018-05-30 02:17:08.647 1 ERROR nova.compute.manager rt.update_available_resource(context, nodename)
2018-05-30 02:17:08.647 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 641, in update_available_resource
2018-05-30 02:17:08.647 1 ERROR nova.compute.manager resources = self.driver.get_available_resource(nodename)
2018-05-30 02:17:08.647 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5892, in get_available_resource
2018-05-30 02:17:08.647 1 ERROR nova.compute.manager disk_over_committed = self._get_disk_over_committed_size_total()
2018-05-30 02:17:08.647 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7393, in _get_disk_over_committed_size_total
2018-05-30 02:17:08.647 1 ERROR nova.compute.manager config, block_device_info)
2018-05-30 02:17:08.647 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7301, in _get_instance_disk_info_from_config
2018-05-30 02:17:08.647 1 ERROR nova.compute.manager dk_size = disk_api.get_allocated_disk_size(path)
2018-05-30 02:17:08.647 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/disk/api.py", line 156, in get_allocated_disk_size
2018-05-30 02:17:08.647 1 ERROR nova.compute.manager return images.qemu_img_info(path).disk_size
2018-05-30 02:17:08.647 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/images.py", line 57, in qemu_img_info
2018-05-30 02:17:08.647 1 ERROR nova.compute.manager raise exception.DiskNotFound(location=path)
2018-05-30 02:17:08.647 1 ERROR nova.compute.manager DiskNotFound: No disk at /var/lib/nova/instances/f3ed9015-3984-43f4-b4a5-c2898052b47d/disk
And resource tracker is no longer updated. We can find lots of these in the gate.
Note that change Icec2769bf42455853cbe686fb30fda73df791b25 nearly mitigates this, but doesn't because task_state is not set while the instance is awaiting confirm.
Original reported in RH Bugzilla: https:/ /bugzilla. redhat. com/show_ bug.cgi? id=1584315
Tested on OSP12 (Pike), but appears to be still present on master. Should only occur if nova compute is configured to use local file instance storage.
Create instance A on compute X
Resize instance A to compute Y lib/nova/ instances/ <uuid A> renamed to <uuid A>_resize on X
Domain is powered off
/var/
Domain is *not* undefined
On compute X: available_ resource runs as a periodic task get_available_ resource( ) over_committed_ size_total
update_
First action is to update self
rt calls driver.
...calls _get_disk_
...iterates over all defined domains, including the ones whose disks we renamed
...fails because a referenced disk no longer exists
Results in errors in nova-compute.log:
2018-05-30 02:17:08.647 1 ERROR nova.compute. manager [req-bd52371f- c6ec-4a83- 9584-c00c5377ac d8 - - - - -] Error updating resources for node compute- 0.localdomain. : DiskNotFound: No disk at /var/lib/ nova/instances/ f3ed9015- 3984-43f4- b4a5-c2898052b4 7d/disk manager Traceback (most recent call last): manager File "/usr/lib/ python2. 7/site- packages/ nova/compute/ manager. py", line 6695, in update_ available_ resource_ for_node manager rt.update_ available_ resource( context, nodename) manager File "/usr/lib/ python2. 7/site- packages/ nova/compute/ resource_ tracker. py", line 641, in update_ available_ resource manager resources = self.driver. get_available_ resource( nodename) manager File "/usr/lib/ python2. 7/site- packages/ nova/virt/ libvirt/ driver. py", line 5892, in get_available_ resource manager disk_over_committed = self._get_ disk_over_ committed_ size_total( ) manager File "/usr/lib/ python2. 7/site- packages/ nova/virt/ libvirt/ driver. py", line 7393, in _get_disk_ over_committed_ size_total manager config, block_device_info) manager File "/usr/lib/ python2. 7/site- packages/ nova/virt/ libvirt/ driver. py", line 7301, in _get_instance_ disk_info_ from_config manager dk_size = disk_api. get_allocated_ disk_size( path) manager File "/usr/lib/ python2. 7/site- packages/ nova/virt/ disk/api. py", line 156, in get_allocated_ disk_size manager return images. qemu_img_ info(path) .disk_size manager File "/usr/lib/ python2. 7/site- packages/ nova/virt/ images. py", line 57, in qemu_img_info manager raise exception. DiskNotFound( location= path) manager DiskNotFound: No disk at /var/lib/ nova/instances/ f3ed9015- 3984-43f4- b4a5-c2898052b4 7d/disk
2018-05-30 02:17:08.647 1 ERROR nova.compute.
2018-05-30 02:17:08.647 1 ERROR nova.compute.
2018-05-30 02:17:08.647 1 ERROR nova.compute.
2018-05-30 02:17:08.647 1 ERROR nova.compute.
2018-05-30 02:17:08.647 1 ERROR nova.compute.
2018-05-30 02:17:08.647 1 ERROR nova.compute.
2018-05-30 02:17:08.647 1 ERROR nova.compute.
2018-05-30 02:17:08.647 1 ERROR nova.compute.
2018-05-30 02:17:08.647 1 ERROR nova.compute.
2018-05-30 02:17:08.647 1 ERROR nova.compute.
2018-05-30 02:17:08.647 1 ERROR nova.compute.
2018-05-30 02:17:08.647 1 ERROR nova.compute.
2018-05-30 02:17:08.647 1 ERROR nova.compute.
2018-05-30 02:17:08.647 1 ERROR nova.compute.
2018-05-30 02:17:08.647 1 ERROR nova.compute.
2018-05-30 02:17:08.647 1 ERROR nova.compute.
And resource tracker is no longer updated. We can find lots of these in the gate.
Note that change Icec2769bf42455 853cbe686fb30fd a73df791b25 nearly mitigates this, but doesn't because task_state is not set while the instance is awaiting confirm.