After upgrading from Rocky to Stein, nova-compute services fail to start new instances with the following error message:
Failed to allocate the network(s), not rescheduling.
Looking in the nova-compute logs, we also see this:
ERROR nova.virt.libvirt.driver [req-8733cf16-6f89-4664-9595-189dacab8a93 7cbd99b5747146baad20c8c035a64706 b44e356a90d74efcbeea1a4024104337 - default default] [instance: 32c493c4-d88c-4f14-98db-c7af64bf3324] Neutron Reported failure on event network-vif-plugged-60c05a0d-8758-44c9-81e4-754551567be5 for instance 32c493c4-d88c-4f14-98db-c7af64bf3324: NovaException: In shutdown, no new events can be scheduled
And this:
2019-03-22 13:49:01.177 7 ERROR nova.compute.manager [req-8733cf16-6f89-4664-9595-189dacab8a93 7cbd99b5747146baad20c8c035a64706 b44e356a90d74efcbeea1a4024104337 - default default] [instance: 32c493c4-d88c-4f14-98db-c7af64bf3324] Failed to allocate network(s): VirtualInterfaceCreateException: Virtual Interface creation failed
2019-03-22 13:49:01.177 7 ERROR nova.compute.manager [instance: 32c493c4-d88c-4f14-98db-c7af64bf3324] Traceback (most recent call last):
2019-03-22 13:49:01.177 7 ERROR nova.compute.manager [instance: 32c493c4-d88c-4f14-98db-c7af64bf3324] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 2235, in _build_and_run_instance
2019-03-22 13:49:01.177 7 ERROR nova.compute.manager [instance: 32c493c4-d88c-4f14-98db-c7af64bf3324] block_device_info=block_device_info)
2019-03-22 13:49:01.177 7 ERROR nova.compute.manager [instance: 32c493c4-d88c-4f14-98db-c7af64bf3324] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3150, in spawn
2019-03-22 13:49:01.177 7 ERROR nova.compute.manager [instance: 32c493c4-d88c-4f14-98db-c7af64bf3324] destroy_disks_on_failure=True)
2019-03-22 13:49:01.177 7 ERROR nova.compute.manager [instance: 32c493c4-d88c-4f14-98db-c7af64bf3324] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5686, in _create_domain_and_network
2019-03-22 13:49:01.177 7 ERROR nova.compute.manager [instance: 32c493c4-d88c-4f14-98db-c7af64bf3324] destroy_disks_on_failure)
2019-03-22 13:49:01.177 7 ERROR nova.compute.manager [instance: 32c493c4-d88c-4f14-98db-c7af64bf3324] File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-03-22 13:49:01.177 7 ERROR nova.compute.manager [instance: 32c493c4-d88c-4f14-98db-c7af64bf3324] self.force_reraise()
2019-03-22 13:49:01.177 7 ERROR nova.compute.manager [instance: 32c493c4-d88c-4f14-98db-c7af64bf3324] File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-03-22 13:49:01.177 7 ERROR nova.compute.manager [instance: 32c493c4-d88c-4f14-98db-c7af64bf3324] six.reraise(self.type_, self.value, self.tb)
2019-03-22 13:49:01.177 7 ERROR nova.compute.manager [instance: 32c493c4-d88c-4f14-98db-c7af64bf3324] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5665, in _create_domain_and_network
2019-03-22 13:49:01.177 7 ERROR nova.compute.manager [instance: 32c493c4-d88c-4f14-98db-c7af64bf3324] error_callback=self._neutron_failed_callback):
2019-03-22 13:49:01.177 7 ERROR nova.compute.manager [instance: 32c493c4-d88c-4f14-98db-c7af64bf3324] File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2019-03-22 13:49:01.177 7 ERROR nova.compute.manager [instance: 32c493c4-d88c-4f14-98db-c7af64bf3324] return self.gen.next()
2019-03-22 13:49:01.177 7 ERROR nova.compute.manager [instance: 32c493c4-d88c-4f14-98db-c7af64bf3324] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 472, in wait_for_instance_event
2019-03-22 13:49:01.177 7 ERROR nova.compute.manager [instance: 32c493c4-d88c-4f14-98db-c7af64bf3324] error_callback(event_name, instance)
2019-03-22 13:49:01.177 7 ERROR nova.compute.manager [instance: 32c493c4-d88c-4f14-98db-c7af64bf3324] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5624, in _neutron_failed_callback
2019-03-22 13:49:01.177 7 ERROR nova.compute.manager [instance: 32c493c4-d88c-4f14-98db-c7af64bf3324] raise exception.VirtualInterfaceCreateException()
2019-03-22 13:49:01.177 7 ERROR nova.compute.manager [instance: 32c493c4-d88c-4f14-98db-c7af64bf3324] VirtualInterfaceCreateException: Virtual Interface creation failed
During the upgrade process, we send nova containers a SIGHUP to cause them to reload their object version state. Speaking to the nova team in IRC, there is a known issue with this, caused by oslo.service performing a full shutdown in response to a SIGHUP, which breaks nova-compute. There is a patch [1] in review to address this.
The workaround is to restart the nova compute service.
[1] https://review.openstack.org/#/c/641907
Fix proposed to branch: master /review. openstack. org/645614
Review: https:/