tripleo

CS9 wallaby sc10-ovn-provider-standalone job failing consistently with error "libvirt.libvirtError: unsupported configuration: Emulator '/usr/libexec/qemu-kvm' does not support virt type 'kvm'"

Bug #2004141 reported by Pooja Jadhav
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Unassigned
tripleo antelope-1

Bug Description

CS9 wallaby sc10-ovn-provider-standalone job failing consistently with error "libvirt.libvirtError: unsupported configuration: Emulator '/usr/libexec/qemu-kvm' does not support virt type 'kvm'" with below error logs :

VIFOpenVSwitch(active=False,address=fa:16:3e:dc:48:31,bridge_name='br-int',has_traffic_filtering=True,id=5135c0a3-7b83-4c20-8285-24f0313b6276,network=Network(60b16a61-e555-4ef0-b25d-f673aed1cf05),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap5135c0a3-7b')
2023-01-30 00:14:08.113 2 INFO nova.virt.libvirt.driver [req-5056476f-247d-4b6e-aac5-b0c22acd395a 42fb2472750a4df2aadbeb21efcc97ff 5b73245f70704eed845f33f6d8c39406 - default default] [instance: 312700ad-8475-4c55-8196-aeab0f291106] Deleting instance files /var/lib/nova/instances/312700ad-8475-4c55-8196-aeab0f291106_del
2023-01-30 00:14:08.114 2 INFO nova.virt.libvirt.driver [req-5056476f-247d-4b6e-aac5-b0c22acd395a 42fb2472750a4df2aadbeb21efcc97ff 5b73245f70704eed845f33f6d8c39406 - default default] [instance: 312700ad-8475-4c55-8196-aeab0f291106] Deletion of /var/lib/nova/instances/312700ad-8475-4c55-8196-aeab0f291106_del complete
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [req-5056476f-247d-4b6e-aac5-b0c22acd395a 42fb2472750a4df2aadbeb21efcc97ff 5b73245f70704eed845f33f6d8c39406 - default default] [instance: 312700ad-8475-4c55-8196-aeab0f291106] Instance failed to spawn: libvirt.libvirtError: unsupported configuration: Emulator '/usr/libexec/qemu-kvm' does not support virt type 'kvm'
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] Traceback (most recent call last):
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 2640, in _build_resources
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] yield resources
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 2409, in _build_and_run_instance
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] self.driver.spawn(context, instance, image_meta,
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 4193, in spawn
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] self._create_guest_with_network(
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7268, in _create_guest_with_network
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] self._cleanup(
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] self.force_reraise()
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] raise self.value
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7237, in _create_guest_with_network
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] guest = self._create_guest(
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7172, in _create_guest
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] guest = libvirt_guest.Guest.create(xml, self._host)
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/guest.py", line 154, in create
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] LOG.error('Error defining a guest with XML: %s',
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] self.force_reraise()
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] raise self.value
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/guest.py", line 151, in create
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] guest = host.write_instance_config(xml)
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/host.py", line 1203, in write_instance_config
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] domain = self.get_connection().defineXML(xml)
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 190, in doit
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] result = proxy_call(self._autowrap, f, *args, **kwargs)
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 148, in proxy_call
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] rv = execute(f, *args, **kwargs)
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 129, in execute
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] six.reraise(c, e, tb)
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] File "/usr/lib/python3.9/site-packages/six.py", line 709, in reraise
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] raise value
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 83, in tworker
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] rv = meth(*args, **kwargs)
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] File "/usr/lib64/python3.9/site-packages/libvirt.py", line 4467, in defineXML
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] raise libvirtError('virDomainDefineXML() failed')
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106] libvirt.libvirtError: unsupported configuration: Emulator '/usr/libexec/qemu-kvm' does not support virt type 'kvm'
2023-01-30 00:14:08.187 2 ERROR nova.compute.manager [instance: 312700ad-8475-4c55-8196-aeab0f291106]
2023-01-30 00:14:08.195 2 INFO nova.compute.manager [req-5056476f-247d-4b6e-aac5-b0c22acd395a 42fb2472750a4df2aadbeb21efcc97ff 5b73245f70704eed845f33f6d8c39406 - default default] [instance: 312700ad-8475-4c55-8196-aeab0f291106] Terminating instance

Logs :

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-wallaby/65e9aae/logs/undercloud/var/log/containers/nova/nova-compute.log

Failing on tempest tests :

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-wallaby/182373d/logs/undercloud/var/log/tempest/stestr_results.html

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-wallaby/a7f0e7c/logs/undercloud/var/log/tempest/stestr_results.html

Revision history for this message
Ronelle Landy (rlandy) wrote :

This may be related to the switch to run on IBM cloud.
Rerunning with testproject:
https://review.rdoproject.org/r/c/testproject/+/42887

to see if the default vexxhost nodeset will fix this

Revision history for this message
Ronelle Landy (rlandy) wrote :

Here is where this fix went in: https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/integration-pipeline-wallaby-centos9.yaml#L49

If the testproject works - remove the yaml anchor from line 49 and use the dependency anchor only like https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/integration-pipeline-wallaby-centos9.yaml#L51

Revision history for this message
Douglas Viroel (dviroel) wrote :

On IBM cloud we are missing nested kvm enabled.
On vexxhost job was passing with kvm enabled[1]:
 - /sys/module/kvm_amd/parameters/nested:1
On IBM, we don't have see nested kvm flag[2]

[1] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-wallaby/f1fd4f5/logs/undercloud/var/log/extra/cpuinfo.txt.gz
[2] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-wallaby/0f1a002/logs/undercloud/var/log/extra/cpuinfo.txt

Revision history for this message
Ronelle Landy (rlandy) wrote :

https://review.rdoproject.org/r/c/testproject/+/42887 passed so we can go forward with the nodeset fix

Revision history for this message
Pooja Jadhav (poojajadhav22) wrote :

Fix is up and its in gate now : https://review.rdoproject.org/r/c/rdo-jobs/+/46782

Revision history for this message
Marios Andreou (marios-b) wrote :

moving this fix-released after the nodeset update merged /rdo-jobs/+/46782

Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.