There is an interesting sequence of log lines in the compute log for the nova instance (instance-00000001) that is reported in the repeated libvirt logs.
Oct 28 23:45:02.776968 nested-virt-ubuntu-bionic-vexxhost-ca-ymq-1-0021169707 nova-compute[16413]: INFO nova.virt.libvirt.driver [-] [instance: 7a343084-ac33-4dcb-8343-744fe2c1fb26] Instance spawned successfully.
Oct 28 23:45:02.776968 nested-virt-ubuntu-bionic-vexxhost-ca-ymq-1-0021169707 nova-compute[16413]: INFO nova.compute.manager [None req-e609f22f-0c5b-4e75-8fed-2cc5ea78decb admin admin] [instance: 7a343084-ac33-4dcb-8343-744fe2c1fb26] Took 326.09 seconds to spawn the instance on the hypervisor.
Oct 28 23:45:02.777289 nested-virt-ubuntu-bionic-vexxhost-ca-ymq-1-0021169707 nova-compute[16413]: DEBUG nova.compute.manager [None req-e609f22f-0c5b-4e75-8fed-2cc5ea78decb admin admin] [instance: 7a343084-ac33-4dcb-8343-744fe2c1fb26] Checking state {{(pid=16413) _get_power_state /opt/stack/nova/nova/compute/manager.py:1557}}
Oct 28 23:45:02.828879 nested-virt-ubuntu-bionic-vexxhost-ca-ymq-1-0021169707 nova-compute[16413]: DEBUG nova.compute.manager [None req-1efcbd1f-1af9-4e08-8572-e04879fa08fb None None] [instance: 7a343084-ac33-4dcb-8343-744fe2c1fb26] Checking state {{(pid=16413) _get_power_state /opt/stack/nova/nova/compute/manager.py:1557}}
Oct 28 23:45:02.832224 nested-virt-ubuntu-bionic-vexxhost-ca-ymq-1-0021169707 nova-compute[16413]: DEBUG nova.compute.manager [None req-1efcbd1f-1af9-4e08-8572-e04879fa08fb None None] [instance: 7a343084-ac33-4dcb-8343-744fe2c1fb26] Synchronizing instance power state after lifecycle event "Resumed"; current vm_state: building, current task_state: deleting, current DB power_state: 0, VM power_state: 1 {{(pid=16413) handle_lifecycle_event /opt/stack/nova/nova/compute/manager.py:1294}}
Oct 28 23:45:02.865991 nested-virt-ubuntu-bionic-vexxhost-ca-ymq-1-0021169707 nova-compute[16413]: INFO nova.compute.manager [None req-1efcbd1f-1af9-4e08-8572-e04879fa08fb None None] [instance: 7a343084-ac33-4dcb-8343-744fe2c1fb26] During sync_power_state the instance has a pending task (deleting). Skip.
Oct 28 23:45:03.052749 nested-virt-ubuntu-bionic-vexxhost-ca-ymq-1-0021169707 nova-compute[16413]: DEBUG nova.compute.utils [None req-e609f22f-0c5b-4e75-8fed-2cc5ea78decb admin admin] [instance: 7a343084-ac33-4dcb-8343-744fe2c1fb26] Conflict updating instance 7a343084-ac33-4dcb-8343-744fe2c1fb26. Expected: {'task_state': ['spawning']}. Actual: {'task_state': 'deleting'} {{(pid=16413) notify_about_instance_usage /opt/stack/nova/nova/compute/utils.py:429}}
Oct 28 23:45:03.065608 nested-virt-ubuntu-bionic-vexxhost-ca-ymq-1-0021169707 nova-compute[16413]: DEBUG nova.compute.manager [None req-e609f22f-0c5b-4e75-8fed-2cc5ea78decb admin admin] [instance: 7a343084-ac33-4dcb-8343-744fe2c1fb26] Instance disappeared during build. {{(pid=16413) _do_build_and_run_instance /opt/stack/nova/nova/compute/manager.py:2266}}
nova reports that the instance is built successfully and right after it it reports that the instance is disappeared during build.
It seems that the whole libvirt error loop happens after nova resumed the guest in libvirt. The fact that the overall instance boot took more than 300sec is due to slow image download.
Would be good to involve the libvirt maintainers to see if they have any idea what happened on the libvirt side
There is an interesting sequence of log lines in the compute log for the nova instance (instance-00000001) that is reported in the repeated libvirt logs.
Oct 28 23:45:02.776968 nested- virt-ubuntu- bionic- vexxhost- ca-ymq- 1-0021169707 nova-compute[ 16413]: INFO nova.virt. libvirt. driver [-] [instance: 7a343084- ac33-4dcb- 8343-744fe2c1fb 26] Instance spawned successfully. virt-ubuntu- bionic- vexxhost- ca-ymq- 1-0021169707 nova-compute[ 16413]: INFO nova.compute. manager [None req-e609f22f- 0c5b-4e75- 8fed-2cc5ea78de cb admin admin] [instance: 7a343084- ac33-4dcb- 8343-744fe2c1fb 26] Took 326.09 seconds to spawn the instance on the hypervisor. virt-ubuntu- bionic- vexxhost- ca-ymq- 1-0021169707 nova-compute[ 16413]: DEBUG nova.compute. manager [None req-e609f22f- 0c5b-4e75- 8fed-2cc5ea78de cb admin admin] [instance: 7a343084- ac33-4dcb- 8343-744fe2c1fb 26] Checking state {{(pid=16413) _get_power_state /opt/stack/ nova/nova/ compute/ manager. py:1557} } virt-ubuntu- bionic- vexxhost- ca-ymq- 1-0021169707 nova-compute[ 16413]: DEBUG nova.compute. manager [None req-1efcbd1f- 1af9-4e08- 8572-e04879fa08 fb None None] [instance: 7a343084- ac33-4dcb- 8343-744fe2c1fb 26] Checking state {{(pid=16413) _get_power_state /opt/stack/ nova/nova/ compute/ manager. py:1557} } virt-ubuntu- bionic- vexxhost- ca-ymq- 1-0021169707 nova-compute[ 16413]: DEBUG nova.compute. manager [None req-1efcbd1f- 1af9-4e08- 8572-e04879fa08 fb None None] [instance: 7a343084- ac33-4dcb- 8343-744fe2c1fb 26] Synchronizing instance power state after lifecycle event "Resumed"; current vm_state: building, current task_state: deleting, current DB power_state: 0, VM power_state: 1 {{(pid=16413) handle_ lifecycle_ event /opt/stack/ nova/nova/ compute/ manager. py:1294} } virt-ubuntu- bionic- vexxhost- ca-ymq- 1-0021169707 nova-compute[ 16413]: INFO nova.compute. manager [None req-1efcbd1f- 1af9-4e08- 8572-e04879fa08 fb None None] [instance: 7a343084- ac33-4dcb- 8343-744fe2c1fb 26] During sync_power_state the instance has a pending task (deleting). Skip. virt-ubuntu- bionic- vexxhost- ca-ymq- 1-0021169707 nova-compute[ 16413]: DEBUG nova.compute.utils [None req-e609f22f- 0c5b-4e75- 8fed-2cc5ea78de cb admin admin] [instance: 7a343084- ac33-4dcb- 8343-744fe2c1fb 26] Conflict updating instance 7a343084- ac33-4dcb- 8343-744fe2c1fb 26. Expected: {'task_state': ['spawning']}. Actual: {'task_state': 'deleting'} {{(pid=16413) notify_ about_instance_ usage /opt/stack/ nova/nova/ compute/ utils.py: 429}} virt-ubuntu- bionic- vexxhost- ca-ymq- 1-0021169707 nova-compute[ 16413]: DEBUG nova.compute. manager [None req-e609f22f- 0c5b-4e75- 8fed-2cc5ea78de cb admin admin] [instance: 7a343084- ac33-4dcb- 8343-744fe2c1fb 26] Instance disappeared during build. {{(pid=16413) _do_build_ and_run_ instance /opt/stack/ nova/nova/ compute/ manager. py:2266} }
Oct 28 23:45:02.776968 nested-
Oct 28 23:45:02.777289 nested-
Oct 28 23:45:02.828879 nested-
Oct 28 23:45:02.832224 nested-
Oct 28 23:45:02.865991 nested-
Oct 28 23:45:03.052749 nested-
Oct 28 23:45:03.065608 nested-
nova reports that the instance is built successfully and right after it it reports that the instance is disappeared during build.
It seems that the whole libvirt error loop happens after nova resumed the guest in libvirt. The fact that the overall instance boot took more than 300sec is due to slow image download.
Would be good to involve the libvirt maintainers to see if they have any idea what happened on the libvirt side