VirtualInterfaceCreateException due to "Timeout waiting for [('network-vif-plugged'..."

Bug #1944779 reported by melanie witt
90
This bug affects 16 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Confirmed
Low
Unassigned
OpenStack Neutron API Charm
New
Undecided
Unassigned
neutron
New
Undecided
Unassigned

Bug Description

Seeing this occasionally in the gate recently, instance spawn fails with VirtualInterfaceCreateException after timing out 300 seconds waiting for the 'network-vif-plugged' event from neutron. The event never arrives (before or after we begin waiting).

Here's log excerpts showing how it looks [1]:

Sep 23 06:10:44.144477 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: DEBUG nova.compute.manager [None req-6d0b7dcf-7f96-47d2-8673-626abdd275b7 tempest-TestSnapshotPattern-20482801 tempest-TestSnapshotPattern-20482801-project] [instance: 487a3a37-fe41-447d-bc77-9ab3fce47473] Preparing to wait for external event network-vif-plugged-9129f5b9-2940-44c2-8d44-04c46df5fe49 {{(pid=110797) prepare_for_instance_event /opt/stack/nova/nova/compute/manager.py:280}}

[...]

Sep 23 06:15:44.961239 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: WARNING nova.virt.libvirt.driver [None req-6d0b7dcf-7f96-47d2-8673-626abdd275b7 tempest-TestSnapshotPattern-20482801 tempest-TestSnapshotPattern-20482801-project] [instance: 487a3a37-fe41-447d-bc77-9ab3fce47473] Timeout waiting for [('network-vif-plugged', '9129f5b9-2940-44c2-8d44-04c46df5fe49')] for instance with vm_state building and task_state spawning: eventlet.timeout.Timeout: 300 seconds

Logstash query:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Instance%20failed%20to%20spawn%3A%20nova.exception.VirtualInterfaceCreateException%3A%20Virtual%20Interface%20creation%20failed%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d

[1] https://paste.opendev.org/show/809546

melanie witt (melwitt)
tags: added: libvirt network
Revision history for this message
melanie witt (melwitt) wrote :
Revision history for this message
Sylvain Bauza (sylvain-bauza) wrote :

Looks to me legit.

Changed in nova:
status: New → Confirmed
importance: Undecided → Low
Revision history for this message
Michal Arbet (michalarbet) wrote :

Hi, I've already filled LP bug here -> https://bugs.launchpad.net/nova/+bug/1951720 , it looks like it's same issue, isn't it ?

Have someone look deeper into this issue ?

Thanks,
Michal

Revision history for this message
Michal Arbet (michalarbet) wrote :

+ in https://bugs.launchpad.net/nova/+bug/1951623 vif-plugged-in also mentioned.

Revision history for this message
Michal Arbet (michalarbet) wrote :

+ Adding logs

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :
Revision history for this message
Michal Arbet (michalarbet) wrote :
Download full text (7.0 KiB)

from Balazs Gibizer <email address hidden>

gibizer@riverbed:~/tmp/LOGS [000] [11:12:49]
$ egrep -e "instance: 09eff2ce-356f-430f-ab30-5de58f58d698.* Timeout waiting for|network-vif-plugged-0761ff2f-ff6a-40a5-85b7-c9b40386cbe0|network-vif-plugged-e64ddd6e-3128-44ee-afd6-07d13643a4be|network-vif-plugged-bf0f9dd2-4fd3-4028-a3dc-e2581ff081d0|network-vif-plugged-8a472e87-4a7a-4ad4-9fbb-fd9785136611|instance: 09eff2ce-356f-430f-ab30-5de58f58d698.* nova.exception.VirtualInterfaceCreateException" -R
compute0/logs/nova/nova-compute.log:2021-11-25 08:50:24.237 8 DEBUG nova.compute.manager [req-6eab0f2a-d7f4-4251-96f4-177fc0c62212 647a328aef9b4a8b890f102d8f018576 49231caa42f34da8a9d10229b5c7a5d8 - default default] [instance: 09eff2ce-356f-430f-ab30-5de58f58d698] Preparing to wait for external event network-vif-plugged-0761ff2f-ff6a-40a5-85b7-c9b40386cbe0 prepare_for_instance_event /var/lib/kolla/venv/lib/python3.9/site-packages/nova/compute/manager.py:280
compute0/logs/nova/nova-compute.log:2021-11-25 08:50:24.238 8 DEBUG nova.compute.manager [req-6eab0f2a-d7f4-4251-96f4-177fc0c62212 647a328aef9b4a8b890f102d8f018576 49231caa42f34da8a9d10229b5c7a5d8 - default default] [instance: 09eff2ce-356f-430f-ab30-5de58f58d698] Preparing to wait for external event network-vif-plugged-e64ddd6e-3128-44ee-afd6-07d13643a4be prepare_for_instance_event /var/lib/kolla/venv/lib/python3.9/site-packages/nova/compute/manager.py:280
compute0/logs/nova/nova-compute.log:2021-11-25 08:50:24.240 8 DEBUG nova.compute.manager [req-6eab0f2a-d7f4-4251-96f4-177fc0c62212 647a328aef9b4a8b890f102d8f018576 49231caa42f34da8a9d10229b5c7a5d8 - default default] [instance: 09eff2ce-356f-430f-ab30-5de58f58d698] Preparing to wait for external event network-vif-plugged-bf0f9dd2-4fd3-4028-a3dc-e2581ff081d0 prepare_for_instance_event /var/lib/kolla/venv/lib/python3.9/site-packages/nova/compute/manager.py:280
compute0/logs/nova/nova-compute.log:2021-11-25 08:50:24.241 8 DEBUG nova.compute.manager [req-6eab0f2a-d7f4-4251-96f4-177fc0c62212 647a328aef9b4a8b890f102d8f018576 49231caa42f34da8a9d10229b5c7a5d8 - default default] [instance: 09eff2ce-356f-430f-ab30-5de58f58d698] Preparing to wait for external event network-vif-plugged-8a472e87-4a7a-4ad4-9fbb-fd9785136611 prepare_for_instance_event /var/lib/kolla/venv/lib/python3.9/site-packages/nova/compute/manager.py:280
compute0/logs/nova/nova-compute.log:2021-11-25 08:51:00.817 8 DEBUG nova.compute.manager [req-310bc640-20d2-4ff8-8ab0-7a8ed6a49b32 01a75e3a9a9148218916d3beafae2120 7d16babef0524b7dade9a59b0a3569e1 - default default] [instance: 09eff2ce-356f-430f-ab30-5de58f58d698] Received event network-vif-plugged-0761ff2f-ff6a-40a5-85b7-c9b40386cbe0 external_instance_event /var/lib/kolla/venv/lib/python3.9/site-packages/nova/compute/manager.py:10439
compute0/logs/nova/nova-compute.log:2021-11-25 08:51:00.819 8 DEBUG nova.compute.manager [req-310bc640-20d2-4ff8-8ab0-7a8ed6a49b32 01a75e3a9a9148218916d3beafae2120 7d16babef0524b7dade9a59b0a3569e1 - default default] [instance: 09eff2ce-356f-430f-ab30-5de58f58d698...

Read more...

Revision history for this message
Felipe Alencastro (falencastro) wrote :

Just had this same issue on charmed Xena, and managed to workaround it by following Tolga Kaprol's suggestion at https://discourse.charmhub.io/t/community-opinion-on-reliability-and-success-rate-of-ussuri-deployments/3510/3

Douglas Viroel (dviroel)
Changed in nova:
status: Confirmed → Opinion
status: Opinion → Confirmed
Revision history for this message
Alok Prudhvi (alokprudhvi) wrote :

Hello, I'm seeing the same issue where nova error message tells - nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed and the instance creation keeps failing i was able to workaround this by Tolga Kaprol's comments but can i know's causing the exact issue, below are the versions i'm using.

OpenStack Release - Yoga
Nova Version - 25.0.0
Neutron Version - 20.1.0
Libvirt Version - libvirtd (libvirt) 8.0.0

Can someone pls look into and see what's the exact issue.

tags: added: sts
tags: removed: sts
Revision history for this message
Pedro Victor Lourenço Fragola (pedrovlf) wrote :
Revision history for this message
yehaifeng (yehaifeng) wrote :

I have the same problem.

Any method to solve?

Revision history for this message
Alok Prudhvi (alokprudhvi) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.