cpu features hle and rtm disabled for security are present in /usr/share/libvirt/cpu_map.xml

Hi Dave,
IIRC Openstack either tries to determine the least common denominator (in cpu features) or whatever you pass to hi, in your case that was:
[libvirt]
cpu_mode = custom
cpu_model = Skylake-Server-IBRS

And your guest definition won't change after the initial definition. Even if you would run host-model instead of a named type it would (in the past) have determined the hle and rtm features and now can't start with them.

But Skylake-Server-IBRS is a name for a defined set of feature and it would be a bug to change "Skylake-Server-IBRS" to now contain other features.

As you have spotted yourself people could set tsx=yes on the commandline or for whatever probably non-smart reason run with a kernel without the fixes.

Therefore changing the existing "Skylake-Server-IBRS" is a no-go as an SRU, lets consider other options.

---

Upstream did create these new custom names with the -IBRS suffix when the first security issues hit. But as you know there were many issues following that one like L1TF, MDS, ....
Upstream realized quickly that this would be a massive type proliferation that grows even further every now and then.

Also these types back then got defined in qemu not libvirt, you can see them with
  $ qemu-system-x86_64 -cpu ?
Libvirt only tracks names and features of those in /usr/share/libvirt/cpu_map*.

---

Interestingly for all the dangers and drawbacks of host-passthrough, in these cases those setups would not care as they would just pass less features. But modelling the features in libvirt or openstack made them explicit and it is now correctly telling us that it can't provide those.

---

Back when the first set of spectre mitigations hit Daniel made a great post summarizing how configuring models&features works including modifying the named models to yoour needs.
=> https://www.berrange.com/posts/2018/06/29/cpu-model-configuration-for-qemu-kvm-on-x86-hosts/
An example for your "Skylake-Server-IBRS" would now be in libvirt like:
   <cpu mode='custom'>
       <model>Skylake-Server-IBRS</model>
       <feature name="hle" policy="disable"/>
       <feature name="rtm" policy="disable"/>
   </cpu>

Therefore from libvirt's perspective there isn't much we can/should do IMHO, I'll double check if upstream on qemu/libvirt considered otherwise and again defined new types or other quirks. But looking at L1TF, MDS and such I'm expecting that using individual features is what is expected.

---

Lets summarize the options we have right now:

a) You can define your own types for libvirt in /usr/share/libvirt/cpu_map, that seems tempting at first, but
  a1) you'd still need to change the type in every guest, so you gained nothing
  a2) those are not meant to be edited, e.g. they are no conffiles and will
      be overwritten on upgrades of libvirt0
b) Define a new type in qemu and then libvirt as the -IBRS types
  b1) as I said recent security fixes didn't do this anymore, I don't expect this to be different
  b2) this needs to be in sync with others (upstream and distros) or proliferation
      and confusion gets even worse
c) Start to define your guests based on feature and not (only) on names
  c1) that is what most recent security fixes ...

I found no changes since the CRD nor a discussion on the ML of qemu or libvirt.
There are the backported fixes that add taa-no and pschange-mc-no to qemu - but nothing touches hle/rtm yet.

The general answer of of the virt stack avoiding type proliferation are versioned CPU models.
=> https://git.qemu.org/?p=qemu.git;a=commit;h=aa5b969287125d1924d74648b378d4abba544465
=> https://git.qemu.org/?p=qemu.git;a=commit;h=d86a708815c3bec0b934760e6bdab7eb647087b8

But that is a feature that will be in qemu >4.1 and newer libvirt, so look towards Ubuntu 20.04 and UCA derived from that for this feature.
It seems way too big and invasive for an SRU.
But even if you had that, you'd either have
a) unversioned type without consistency (might change on update, which isn't what you want either)
b) versioned types which stay static behave exactly as what we have now

@security - where there discussions about how to handle these feature losses in the "closed circle" around the TSX/TAA bug?

@UCA-Team
I'm also subscribing the UCA Team as I personally only know about [1] which allows you to use named CPU models.

I expect there already is some general pattern established to handle e.g. the older MDS which if you look at "Configuration as a Hypervisor" at [2] also needs such <feature...> entries to be configured through libvirt.
Eventually dropping hle/rtm is no different than adding md-clear back then.

@UCATeam - How would in a OS/Charm world one add/disable individual cpu features?
Could you outline how this is handled today so we might consider adapting the same for hle/rtm?

[1]: https://wiki.openstack.org/wiki/LibvirtXMLCPUModel
[2]: https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/MDS

There's a related fix for enabling CPU feature flags that landed in upstream nova and charm-nova-compute via LP: #1750829 as a result of Meltdown.

There's mention in the upstream fix [1] that a future patch will allow disabling of CPU feature flag but I'm not sure if that has landed. I'll dig some more to see.

[1] https://review.opendev.org/#/c/534384/

Fyi the config for the nova-compute charm that corresonds to this change is cpu-model-extra-flags, defined as a space delimited list of specific CPU flags for libvirt/

I chatted with Kashyap in #openstack-nova and he has a blueprint up for disabling CPU feature flags at: https://blueprints.launchpad.net/nova/+spec/allow-disabling-cpu-flags

Kashyap mentioned an option for now: One (very valid) 'workaround' is that have QEMU add new "named CPU models" to remove the said flags. For that, upstream QEMU folks must add them...please file a QEMU "RFE" bug on launchpad for it.

@Cpaelzer, thoughts on that ^ ?

@Cpaelzer there were alas no discussions pre-disclosure beyond adding the basic taa-no and pschange-mc-no flag support to qemu.

We can definitely add text to the USN about this; let's get it in the KnowledgeBase article first since that's easier to modify.

A small addendum to what Corey said: Upstream QEMU will mostly providing new named CPU models with 'hle' and 'rtm' CPU flags turned off.

Keep an eye on the upstream 'qemu-devel' list :-)

Patches are in progress for new CPU models with TSX disabled: https://lists.gnu.org/archive/html/qemu-devel/2019-11/msg03323.html (Thanks Cpaelzer and Kashyap)

This was now accepted upstream in qemu.
[1] is the merge commit containing the new names discussed here 02fa60d1 / 9ab2237f but also tsx-ctrl which seems to be part of the overall tsx handlign and not yet part of the patches of last week 2a9758c5.

@security - are you gonna take a look at picking those up?

P.S. I haven't seen similar updates to libvirt CPU maps yet.

[1]: https://git.qemu.org/?p=qemu.git;a=commit;h=2061735ff09f9d5e67c501a96227b470e7de69b1

Subscribed ~field-medium

FYI: I got message from sbeattie that security will take a look at the backports for these qemu fixes.

For libvirt we can check again then if the types got updated, and if no one did I can send something there as well.

I'm not sure I'm following the discussion here, but I see hle and rtm flags with the latest security update for bionic GA kernel. So looks like the issue is not reproducible any longer. Am I missing something?

$ dpkg -l | grep linux-image
ii linux-image-4.15.0-72-generic 4.15.0-72.81 amd64 Signed kernel image generic
ii linux-image-generic 4.15.0.72.74 amd64 Generic Linux kernel image

$ uname -a
Linux <hostname> 4.15.0-72-generic #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
...
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6150 CPU @ 2.70GHz
Stepping: 4
...
Virtualization: VT-x
...
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_pkg_req pku ospke md_clear flush_l1d

You said this might have been resolved differently anyway with the newest kernel having again hle/rtm enabled - I haven't heard about it but that would probably be even better.
Lets see on the kernel side.
- Fixes for CVE-2019-11135 got added in 4.15.0-69.78
- This was reported against 4.15.0-70
- Wondering about 4.15.0-72 being ok again

Reading the latest state of Documentation/admin-guide/hw-vuln/tsx_async_abort.rst shows:
https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/tsx_async_abort.html#mitigation-control-on-the-kernel-command-line

212 tsx=on tsx_async_abort=full The system will use VERW to clear CPU
213 buffers. Cross-thread attacks are still
214 possible on SMT machines.
215 tsx=on tsx_async_abort=full,nosmt As above, cross-thread attacks on SMT
216 mitigated.
217 tsx=on tsx_async_abort=off The system is vulnerable.
218 tsx=off tsx_async_abort=full TSX might be disabled if microcode
219 provides a TSX control MSR. If so,
220 system is not vulnerable.
221 tsx=off tsx_async_abort=full,nosmt Ditto
222 tsx=off tsx_async_abort=off ditto

Maybe the initial take was tsx=off which would switch off those flags. But now is any of the tsx=on but with full mitigations? But I'm guessing at this point.
I have not found a clear kernel change since then (not until 4.15.0-73.82, but even less so between .70 and .72) that would change these.
The only related "- x86/speculation/taa: Fix printing of TAA_MSG_SMT on IBRS_ALL CPUs" seems to only affect print output, but not change behavior.

Furthermore none of the systems I have has got hle/rtm back since then.

@Nobuto - has your system any of the above kernel parameters set manually?

I haven't heard from this by sbeattie or others after my last update.
Lets ping security to be sure this hasn't been forgotten.
(I have done that on IRC as well)
@Security - any updates on this from your side?

> @Nobuto - has your system any of the above kernel parameters set manually?

No, we don't have any flags in kernel parameters related to tsx or similar.

FWIW, I haven't tested any older kernel to check if those flags are available. But we are using Intel(R) Xeon(R) Gold 6150 CPU @ 2.70GHz.

@Nobuto:
Interesting what is your:
grep . /sys/devices/system/cpu/vulnerabilities/*

Here you are:

$ grep . /sys/devices/system/cpu/vulnerabilities/*
/sys/devices/system/cpu/vulnerabilities/itlb_multihit:KVM: Mitigation: Split huge pages
/sys/devices/system/cpu/vulnerabilities/l1tf:Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable
/sys/devices/system/cpu/vulnerabilities/mds:Mitigation: Clear CPU buffers; SMT vulnerable
/sys/devices/system/cpu/vulnerabilities/meltdown:Mitigation: PTI
/sys/devices/system/cpu/vulnerabilities/spec_store_bypass:Mitigation: Speculative Store Bypass disabled via prctl and seccomp
/sys/devices/system/cpu/vulnerabilities/spectre_v1:Mitigation: usercopy/swapgs barriers and __user pointer sanitization
/sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: Full generic retpoline, IBPB: conditional, IBRS_FW, STIBP: conditional, RSB filling
/sys/devices/system/cpu/vulnerabilities/tsx_async_abort:Mitigation: Clear CPU buffers; SMT vulnerable

Oh so you seem to have a combination of HW/FW/Kernel that gets along with it. But for e.g. a Hosting environment please be aware of the "SMT vulnerable" unless you don't have SMT disabled anyway.
Since different systems (HW/FW/Kernel) will behave differently I think this issue isn't resolved yet @Nobuto.

@Security please let us know what your plan is on that - backporting the named types with the features removed to qemu/libvirt or did this change overall.

Hi,

There is also the microcode package where Intel has published and updated these flags. So this in combination with the kernel might cause these errors.

-J

FYI: I pinged the security Team again on this one to be sure it doesn't fall through the cracks

These look like the list of commits to support recent kernel/microcode feature updates:

qemu:
https://git.qemu.org/?p=qemu.git;a=commit;h=7fac38635e1cc5ebae34eb6530da1009bd5808e4 (taa)
https://git.qemu.org/?p=qemu.git;a=commit;h=0723cc8a5558c94388db75ae1f4991314914edd3 (vmx)
https://git.qemu.org/?p=qemu.git;a=commit;h=2a9758c51e2c2d13fc3845c3d603c11df98b8823 (tsx)
https://git.qemu.org/?p=qemu.git;a=commit;h=9ab2237f1979f31de228b2a73b56925dbde938d1 (tsx)
https://git.qemu.org/?p=qemu.git;a=commit;h=02fa60d10137ed2ef17534718d7467e0d2170142 (tsx)
https://git.qemu.org/?p=qemu.git;a=commit;h=c6f3215ffa3064fd04e00d2b159c3b90c3c9b1a5 (vmx)

libvirt:
https://libvirt.org/git/?p=libvirt.git;a=commit;h=07aaced4e6ea6db8b27f44636f51cafa6f1847a8 (taa)
https://libvirt.org/git/?p=libvirt.git;a=commit;h=f411b7ef68221e82dec0129aaf2f2a26a8987504 (tsx)

I don't know what the way forward is to resolve this issue. While upstream qemu has added some new CPU models, "Skylake-Client-noTSX-IBRS", "Skylake-Server-noTSX-IBRS", etc, libvirt has not. If I do add these to libvirt, we will need to carry them forward as a delta to upstream possibly forever.

Even adding those new CPU models is just a workaround that makes manually changing the CPU model easier to do, but does not present a solution for this issue. There is no automatic way to fix this issue that wouldn't cause migration failures.

FYI: bug 1861643 might be a new symptom of the same root cause. If that is confirmed we might bump this for re-considering our options ... again (at least going forward we might want to add Skylake-Client-noTSX-IBRS and such to qemu in Focal).

.. or adding the types to libvirt, whatever is needed to at least limit the impact of dropping hle/rtm in 20.04 ...

Upstream agrees, we need to add new types to libvirt.
The Discussion is still ongoing but I'm prepping a submission of those types ...

List of types to consider adding by comparing qemu qmp output of cpu probing with cpu_maps of libvirt:

-Cascadelake-Server-noTSX
-Icelake-Client-noTSX
-Icelake-Server-noTSX
-Skylake-Server-noTSX-IBRS
-Skylake-Client-noTSX-IBRS

Submitted to upstream libvirt as:
=> https://www.redhat.com/archives/libvir-list/2020-March/msg00175.html

FYI: discussion goes on, v2 now submitted to the thread on the ML

Now reply yet on v2:
https://www.redhat.com/archives/libvir-list/2020-March/msg00296.html

FYI I pinged on the list as I see that "auto-detect other cpu types" not linked to the submission I made. But OTOH we can't go ahead integrating this in Ubuntu ahead of time while not accepted or we would risk to diverge making things even worse ...

diverge ... details of names CPU types I mean, breaking cross system and cross release behavior

Updated task state to reflect my work with upstream to get libvirt to know about this change.

PPA: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3986
MP: https://code.launchpad.net/~paelzer/ubuntu/+source/libvirt/+git/libvirt/+merge/380942

Tests with the PPA build using the patches sent by Jiri (and my addition of the noTSX types):

#1 virsh capabilities
Before:
Broadwell-noTSX-IBRS + 33 features

After:
Skylake-Client-noTSX-IBRS +24 features

=> good

#2 virsh domcapabilities
Before:
Skylake-Client-IBRS + 16 features

After:
Skylake-Client-IBRS + 16 features (unchanged as intended for compatibility)

#3 usable models
Before:
only older types

After:
now added "Skylake-Client-noTSX-IBRS" which is a more modern IBRS type than the others I had

My system isn't new enough to get the others added, but that is fine as a test.
Also the type "Skylake-Server-noTSX-IBRS" worked, auto-disabling the avx features my chip is missing.

I started a guest with such a type through libvirt and it looks as expected:
  -cpu Skylake-Client-noTSX-IBRS

#4 and finally the adapted tests still ran fien at build time.

FYI Regression test is fine on this build, but waiting a bit to give it a chance to be upstream committed

Since tests are good I asked if they are ready to be committed and got:
  [13:31] <jdenemar> cpaelzer_: I'll commit it later today

With that I can later on replace the preliminary patches with the final ones before an upload.

Upstream committed as:
dd17a4eba8 cpu_map: Add more -noTSX x86 CPU models
f4914045c2 cpu_map: Add <decode> element to x86 CPU model definitions
7cd896ef31 cpu_x86: Honor CPU models' <decode> element
17cdefe5f1 cpu_map: Don't use new noTSX models for host-model CPUs

I have replaced my patches with the final ones which comes down to replacing the link to the upstream commit from an ML entry to a commit.

This bug was fixed in the package libvirt - 6.0.0-0ubuntu6

---------------
libvirt (6.0.0-0ubuntu6) focal; urgency=medium

  * d/p/ubuntu/lp-1867460-*: fix domcapabilities before capabilities
    and binary autodetection in general (LP: #1867460)
  * d/p/stable/lp-1868539-*: stabilize libvirt by backporting upstream
    fixes (LP: #1868539)
  * d/p/ubuntu/lp-1853200*: add cpu models without hle/rtm features to have
    modern types on kernels with recent security fixes (LP: #1853200)
  * d/p/ubuntu/lp-1868528-*: Fail when fetching CPU Status for invalid CPU
    (LP: #1868528)

 -- Christian Ehrhardt <email address hidden> Fri, 20 Mar 2020 10:34:19 +0100

Will these changes go into 18.04.x also?

-J

A while back Marc checked the case and realize that backporting the qemu changes without anything in libvirt would make no sense - comment #21.
Now the things in libvirt exist, which is a step forward.

This still will be no transparent solution, people will have to switch types if they can't run the old types. And it appears that if one is willing to change the cpu-model, he could also add feature-disable hle/rtm with the same effort.

The special case is for situations in which people can select cpu-models but not define custom features - for those backporting these would help to mitigate the impact of the CVE related TSX/TAA kernel changes that started all of this.

I'd suggest to let this mature in focal for a few days, see if people or tests run into issues. And then ask Marc to re-evaluate again.
I'll add qemu/libvirt backport tasks and assign them to Ubuntu security - so that they can comment on what the think (now with the libvirt changes existing).

A quick workaround for those who need hle and rtm CPU flags back is to set the tsx=on kernel boot parameter:

# /etc/default/grub
...
GRUB_CMDLINE_LINUX_DEFAULT="... tsx=on"
...

update-grub

Tested on Ubuntu 20.04 with kernel 5.4.0-31-generic:
# uname -a
Linux com1-dev 5.4.0-31-generic #35-Ubuntu SMP Thu May 7 20:20:34 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

# lscpu | grep Flags
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm arat pln pts pku ospke avx512_vnni md_clear flush_l1d arch_capabilities

The Eoan Ermine has reached end of life, so this bug will not be fixed for that release

Status changed to 'Confirmed' because the bug affects multiple users.