OpenStack Compute (nova)

evmc not supported on AMD processors but is enabled by default for "windows" instances

Bug #2009280 reported by Forrest Fuqua
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Triaged
High
Unassigned

Bug Description

on AMD Processors it is known that QEMU does not not support hv-evmc
but as of notes/update-libvirt-enlightenments-for-windows-23abea98cc1db667.yaml it was enabled

There is no way that I can see to single out this flag and nova should check if the remote CPU is AMD and disable this enablement.

You can test this on any AMD machine with "qemu-system-x86_64 --nographic -cpu host,hv_relaxed,hv_vapic,hv_spinlocks=8191,hv_vpindex,hv_runtime,hv_synic,hv_stimer,hv_reset,hv_frequencies,hv_time,hv_evmcs --enable-kvm" and it will throw a critical error of "Hyper-V enlightened VMCS (hv-evmcs) is not supported by kernel"

Here are example logs that are thrown by nova when trying to start the instance, the instance was set to os_type: windows to have it throw this. This was tested on Zed with a AMD EYPC 7702

2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [None req-2172765f-0769-4b82-88fa-3cd68144ee42 ff259b03efc14bee83c881bc4920a12d a69aa169729a4dbea0d9b9683072260d - - default default] [instance: d416180c-cd56-4830-9787-3bb4953d9b02] Failed to build and run instance: libvirt.libvirt]
2023-03-04T20:57:46.694416Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.tsc-scale [bit 4]
2023-03-04T20:57:46.694441Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.vmcb-clean [bit 5]
2023-03-04T20:57:46.694465Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.pause-filter [bit 10]
2023-03-04T20:57:46.694489Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.pfthreshold [bit 12]
2023-03-04T20:57:46.694513Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.svme-addr-chk [bit 28]
2023-03-04T20:57:46.694549Z qemu-kvm: Hyper-V enlightened VMCS (hv-evmcs) is not supported by kernel
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] Traceback (most recent call last):
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/compute/manager.py", line 2517, in _build_and_run_instance
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] self.driver.spawn(context, instance, image_meta,
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 4366, in spawn
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] self._create_guest_with_network(
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7724, in _create_guest_with_network
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] self._cleanup(
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] self.force_reraise()
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] raise self.value
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7701, in _create_guest_with_network
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] guest = self._create_guest(
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7640, in _create_guest
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] guest.launch(pause=pause)
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/virt/libvirt/guest.py", line 168, in launch
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] LOG.exception('Error launching a defined domain with XML: %s',
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] self.force_reraise()
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] raise self.value
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/virt/libvirt/guest.py", line 165, in launch
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] return self._domain.createWithFlags(flags)
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 193, in doit
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] result = proxy_call(self._autowrap, f, *args, **kwargs)
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 151, in proxy_call
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] rv = execute(f, *args, **kwargs)
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 132, in execute
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] six.reraise(c, e, tb)
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/six.py", line 719, in reraise
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] raise value
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 86, in tworker
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] rv = meth(*args, **kwargs)
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/usr/lib64/python3.9/site-packages/libvirt.py", line 1398, in createWithFlags
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] raise libvirtError('virDomainCreateWithFlags() failed')
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2023-03-04T20:57:46.694375Z qemu-kvm: warning: This feature depends on other features that were not request]
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] 2023-03-04T20:57:46.694416Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.tsc-scale [bit 4]
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] 2023-03-04T20:57:46.694441Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.vmcb-clean [bit 5]
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] 2023-03-04T20:57:46.694465Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.pause-filter [bit 10]
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] 2023-03-04T20:57:46.694489Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.pfthreshold [bit 12]
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] 2023-03-04T20:57:46.694513Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.svme-addr-chk [bit 28]
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] 2023-03-04T20:57:46.694549Z qemu-kvm: Hyper-V enlightened VMCS (hv-evmcs) is not supported by kernel
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]
2023-03-04 15:57:47.726 7 INFO os_vif [None req-2172765f-0769-4b82-88fa-3cd68144ee42 ff259b03efc14bee83c881bc4920a12d a69aa169729a4dbea0d9b9683072260d - - default default] Successfully unplugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:23:e5:4f,bridge_name='br-int',has_traffic)
2023-03-04 15:57:47.981 7 INFO nova.compute.manager [None req-2172765f-0769-4b82-88fa-3cd68144ee42 ff259b03efc14bee83c881bc4920a12d a69aa169729a4dbea0d9b9683072260d - - default default] [instance: d416180c-cd56-4830-9787-3bb4953d9b02] Took 0.26 seconds to deallocate network for instan.
2023-03-04 15:57:48.060 7 INFO nova.scheduler.client.report [None req-2172765f-0769-4b82-88fa-3cd68144ee42 ff259b03efc14bee83c881bc4920a12d a69aa169729a4dbea0d9b9683072260d - - default default] Deleted allocations for instance d416180c-cd56-4830-9787-3bb4953d9b02
2023-03-04 15:57:48.079 7 INFO nova.compute.manager [-] [instance: d416180c-cd56-4830-9787-3bb4953d9b02] During sync_power_state the instance has a pending task (block_device_mapping). Skip.

Tags: libvirt
Forrest Fuqua (fffics)
tags: added: libvirt
tags: added: hyper-v
Revision history for this message
sean mooney (sean-k-mooney) wrote :

there are no references to this being a intel exclucive featur in the libvirt docs

https://libvirt.org/formatdomain.html#hypervisor-features

but it would be relitivly simple to check for amd here

https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L6196

unless you have seen this in the hyper-v virt driver this should just have the libvirt tag

tags: removed: hyper-v
Revision history for this message
sean mooney (sean-k-mooney) wrote :

triaging as medium as this will only affect a subset of deplyments where amd host are used in combination with images that decalre the os_type as windows.

this can be worked around by not setting OS_TYPE=windows which will disable all enlightementes.

Changed in nova:
importance: Undecided → Medium
status: New → Triaged
Revision history for this message
sean mooney (sean-k-mooney) wrote :

i also dont see it documented here

https://www.qemu.org/docs/master/system/i386/hyperv.html

that hv-evmc so i dont think that its widely known that its not supproted on amd.

can you link to where this is documented?

Revision history for this message
Forrest Fuqua (fffics) wrote (last edit ):

It does show in the docs, its just not pointed out very well,

https://www.qemu.org/docs/master/system/i386/hyperv.html

hv-evmcs

    The enlightenment is nested specific, it targets Hyper-V on KVM guests. When enabled, it provides Enlightened VMCS version 1 feature to the guest. The feature implements paravirtualized protocol between L0 (KVM) and L1 (Hyper-V) hypervisors making L2 exits to the hypervisor faster. *The feature is Intel-only.*

    Note: some virtualization features (e.g. Posted Interrupts) are disabled when hv-evmcs is enabled. It may make sense to measure your nested workload with and without the feature to find out if enabling it is beneficial.

    Requires: hv-vapic

Revision history for this message
sean mooney (sean-k-mooney) wrote :

ack there are two ways to fix this.

the simple one would be to just stop enabling it always.

the more complex one is to advertise this capability as a trait and recorded if vms use it or not then schulde based on the trait and its usage.

I'm leaning toward the first option now as a backpackable fix and the second approach as a wishlist feature to do in the future.

Revision history for this message
Sylvain Bauza (sylvain-bauza) wrote :

Putting the bug importance to High as it's preventing a nova boot on a windows-based image on AMD processors (if the image property is set). This is quite a large impact and we would like to fix it sooner than later.

Changed in nova:
importance: Medium → High
Revision history for this message
Ionuț Bîru (ionut-3) wrote :

for me is important to keep os_type=windows in place and for running instances, on compute nodes with amd I had to modify nova/virt/libvirt/driver.py line 6170 to set hv.evmcs = False.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.