kernel: improve spectre mitigation

------- Comment (attachment only) From <email address hidden> 2018-09-03 07:34 EDT-------

------- Comment (attachment only) From <email address hidden> 2018-09-03 07:34 EDT-------

------- Comment From <email address hidden> 2018-09-03 08:09 EDT-------
Following upstream commits

aeaf7002a76c8da60c0f503badcbddc07650678c (kernel 4.19 RC1)
5eda25b10297684c1f46a14199ec00210f3c346e (kernel 4.19 RC1)
c3b9e3e1ea1c1d1524b56b6734711db2a6fc2163 (kernel 4.17 RC1)
a3da7b4a3be51f37f434f14e11e60491f098b6ea (kernel 4.17 RC1)

have to be applied to

1) Ubuntu 18.10 - kernel 4.18
2) Ubuntu 18.04.1 - kernel 4.15
3) Ubuntu 16.04.5 - kernel 4.15
3) Ubuntu 16.04.1 - kernel 4.4

In addition 2 patches are applied

One for 16.04 and one for 18.04.

The extra patch for 16.04 and 18.04 seems to be a follow-up of commit de5cb6eb514ebe241e3edeb290cb41deb380b81d upstream ("s390: use expoline thunks in the BPF JIT"). First question: will that extra change go upstream, too? Second question: There seems to be a related follow-up: 26f8438 "s390: fix br_r1_trampoline for machines without exrl" which is not, yet, included in 18.04 or 16.04 (fwiw neither seems to be in upstream stable 4.4.y). Should that get picked up along with the other changes? And finally (to confirm) this is "CVE-2018-3639 (s390x)" follow-ups?

------- Comment From <email address hidden> 2018-09-04 06:28 EDT-------
The extra patch is a follow-up for de5cb6eb51 ("s390: use expoline thunks in
the BPF JIT"), but the extra patch can not go upstream as the code it patches
is not present anymore. See git commit e1cf4befa2 ("bpf, s390x: remove
ld_abs/ld_ind").

The patch 26f8438 ("s390: fix br_r1_trampoline for machines without exrl")
has just been picked up by Greg for stable-4.4, see [PATCH 4.4 71/80] in
his latest patch series. This patch is only required when building for
machines prio to z10.

All of these patches are related to the original spectre-v2 CVE-2017-5715,
CVE-2018-3639 is speculative store bypass aka spectre variant 4, no ?

Hi Martin,

thanks for the details on the additional patch. For the follow-up, I guess if it goes into 4.4.y it will end up in our 16.04 kernel, even though we probably would not strictly need it there either. And 18.04, too. But in both cases it could be included with the cve update or not. About the CVE number, *sigh* yes, you are right. I could have sworn I was searching for Spectre.v2 and came up with the other number but now I also end up with CVE-2017-5715.

The patches specified above apply easily to the 4.15 (bionic) kernel (with some minor changes), but
it is taking longer to prepare the patches for xenial (4.4) since at least 2 of the above patches need significant changing and/or additional patches in order to be backported to the 4.4 kernel.

Probably also should be mentioned that the patch needing adjustment was "KVM: s390: add etoken support for guests" as it tests on the GISA address in the SIE block which does not yet exist in 4.15. We would drop that if section of the patch.

Actually the patch ("KVM: s390: add etoken support for guests") only needed adjustment because I incorrectly resolved a (minor) conflict when applying it and ended up keeping an unnecessary if() section. After resolving the conflict correctly, no adjustment was needed.

Hello Martin,

The requested upstream fixes don't apply cleanly on top of 16.04 4.4 kernel, and backporting them is not straightforward as we first thought. So far I have identified the following commits as pre-reqs to c3b9e3e1ea1c1d1524b56b6734711db2a6fc2163 alone:

c18719638cc1 KVM: s390: generate facility mask from readable list
d6450dc35e21 KVM: s390: Populate mask of non-hypervisor managed facility bits
bb0258cde7a6 s390/facilities: add helper tool to generate facility lists
35b3fde6203b KVM: s390: wire up bpb feature

I'm afraid we would take some time to backport all the requested fixes and their pre-reqs and get it right.

Would it be possible to provide us the backported patches or a tree with the commits on top of our 16.04 4.4 kernel?

Please advice.

Thank you.

------- Comment From <email address hidden> 2018-09-10 07:03 EDT-------
> c18719638cc1 KVM: s390: generate facility mask from readable list
> d6450dc35e21 KVM: s390: Populate mask of non-hypervisor managed facility bits
> bb0258cde7a6 s390/facilities: add helper tool to generate facility lists
> 35b3fde6203b KVM: s390: wire up bpb feature
>
> I'm afraid we would take some time to backport all the requested fixes and their pre-reqs and get it right.

With the kernel version 4.4 this indeed takes some extra patches. In addition to
that we would need a matching qemu version as well which is probably overkill.

I have talked to Christian Borntr?ger about the KVM aspect and he would be fine
if we do not backport the KVM related patches. The end result is that the KVM
guests would use the expoline mitigation by default, even on machines with
the etoken facility. You can still use the spectre_v2=off kernel parameter to
disable expolines.

That reduces the list of git commits to only two:
aeaf7002a76c8da6 "s390: detect etoken facility"
5eda25b10297684c "s390/lib: use expoline for all bcr instructions"
plus the extra patch.

------- Comment (attachment only) From <email address hidden> 2018-09-10 07:05 EDT-------

------- Comment (attachment only) From <email address hidden> 2018-09-10 07:06 EDT-------

Fixes for 18.04 (Bionic) and 16.04 (Xenial) are now committed to the git repos and will be available on kernel builds for the SRU cycle that's starting this week.

Many thanks kernel team for picking this up on short notice!
Changing status to Fix Committed.

------- Comment From <email address hidden> 2018-09-11 13:18 EDT-------
(In reply to comment #19)
> > c18719638cc1 KVM: s390: generate facility mask from readable list
> > d6450dc35e21 KVM: s390: Populate mask of non-hypervisor managed facility bits
> > bb0258cde7a6 s390/facilities: add helper tool to generate facility lists
> > 35b3fde6203b KVM: s390: wire up bpb feature
> >
> > I'm afraid we would take some time to backport all the requested fixes and their pre-reqs and get it right.
>
> With the kernel version 4.4 this indeed takes some extra patches. In
> addition to
> that we would need a matching qemu version as well which is probably
> overkill.
>
> I have talked to Christian Borntr?ger about the KVM aspect and he would be
> fine
> if we do not backport the KVM related patches. The end result is that the KVM
> guests would use the expoline mitigation by default, even on machines with
> the etoken facility. You can still use the spectre_v2=off kernel parameter to
> disable expolines.
>

To write down the obvious.
That is of course only true for the 4.4 kernel. We can drop the KVM etoken and cpumodel patches from 4.4 because it does not yet provide the cpu model. We certainly want to have the KVM patches in the 4.15 kernel (bionic and future HWE)

------- Comment From <email address hidden> 2018-09-11 13:24 EDT-------
For etoken support for KVM guests we also need qemu support to add this to the cpumodel.

The feature is provided by the following upstream commit
https://git.qemu.org/?p=qemu.git;a=commit;h=27e84d4ebd25b981ab27cb590fe06d1b0fcd06d2
commit: 27e84d4ebd25b981ab27cb590fe06d1b0fcd06d2
title: s390x/kvm: add etoken facility

we also need the following commit to make this compile
https://git.qemu.org/?p=qemu.git;a=commit;h=d36f7de82995a42b749c29c5b60ba31483995a42
commit: d36f7de82995a42b749c29c5b60ba31483995a42
title: linux-headers: update

To simplify things, this part of the headers update is sufficient:
https://git.qemu.org/?p=qemu.git;a=blobdiff;f=linux-headers/asm-s390/kvm.h;h=1ab9901911bf55e3aba55488bb20a184d85bbce5;hp=11def143015d572f208bbdf4aa4ecfa094c700a1;hb=d36f7de82995a42b749c29c5b60ba31483995a42;hpb=c61177881cbda50704207dd9fb4811659bbf913e

Please tell me if you want a backport patch for both upstream commits or if you prefer to do the backport yourself. (Do you want a separate launchpad for qemu or can we handle that here?)

The backport would be necessary for the qemu in bionic and cosmic and all relevant cloud archive versions.

Hi Christian, thanks for the broken down header change!
We can stick to this bug and just add a qemu task to it.

I'll take a look at doing this in cosmic soon (before things close/freeze more) and if working fine there we can take a look at making this available in Bionic's qemu as well.
If I hit anything that makes me feel unsure on the backport I'll reach out to you.

I think with changes being upstream there could be a description added :-)
Is there a TL;DR in your bugzilla that we could move to the bug description on LP?

It is a fix in the means of it would qualify even for an SRU under the "HW exploitation" as well as the "Security" banner. Therefore I think we are ok to push for this right now unless cosmic is fully frozen for the release week or so.

But I'd need to ask you (=IBM) to give the feature some testing as it needs HW not currently available to me.

Could you check the PPA [1] in regard to the change requested here please?

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3413

------- Comment From <email address hidden> 2018-09-12 05:44 EDT-------
i can verify the qemu changes in the ppa.

As discussed you might need the proposed kernel of cosmic.
If you use 4.18.0-8.9 of [1] that should have the etoken changes as you need it.

[1]: https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/proposed

------- Comment From <email address hidden> 2018-09-12 07:44 EDT-------
tested sucessfully with the ppas.

With an old host kernel, qemu reject the etoken facility (and the guest falls back to expoline).
With 4.18.0-8-generic qemu allows etoken and passes that along to the guest.

looks good.

Thanks for the check, especially that you did it with both kernels to be sure on the behavior.
Uploading to cosmic then ...

Ok, cosmic is almost released.
I'm preparing Bionic - could you pre-verify that the backport worked as expected on the ppa at [1] please?
This contains an updated qemu for Bionic.

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3414

This bug was fixed in the package qemu - 1:2.12+dfsg-3ubuntu6

---------------
qemu (1:2.12+dfsg-3ubuntu6) cosmic; urgency=medium

  * improve s390x spectre mitigation with etoken facility (LP: #1790457)
    - debian/patches/ubuntu/lp-1790457-s390x-kvm-add-etoken-facility.patch
    - debian/patches/ubuntu/lp-1790457-partial-s390x-linux-headers-update.patch

 -- Christian Ehrhardt <email address hidden> Wed, 12 Sep 2018 10:06:48 +0200

------- Comment From <email address hidden> 2018-09-13 10:07 EDT-------
that qemu looks good. I had to use an upstream kernel as the kernel patch is still in the pipe.

Thanks for the extra check!

Uploaded to Bionic-unapproved for SRU Team review

Hello bugproxy, or anyone else affected,

Accepted qemu into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:2.11+dfsg-1ubuntu7.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

To be clear - this relies on special HW to be present, so I can't validate it.
IBM was so kind to verify the PPAs in advance, it would be great if you could do so again with the bits in proposed.

------- Comment From <email address hidden> 2018-09-27 10:42 EDT-------
qemu 2.11+dfsg-1ubuntu7.6 from proposed does work as expected.

Thanks for testing, setting tags accordingly.

Since bionic-updates containes kernel 4.15.0-36. and git says that 's390: detect etoken facility' is included:
$ git log --oneline | grep "s390: detect etoken facility"
cffc6b1 s390: detect etoken facility
$ git tag --contains cffc6b1
Ubuntu-4.15.0-35.38
Ubuntu-4.15.0-36.39
Ubuntu-4.15.0-37.40
Ubuntu-raspi2-4.15.0-1025.27
I'm marking bionic as Fix Released.

Since cosmic contains kernel 4.18.0.8.9 and git says that 's390: detect etoken facility' is included:
$ git log --oneline | grep "s390: detect etoken facility"
edb9bc2 s390: detect etoken facility
$ git tag --contains edb9bc2
Ubuntu-4.18.0-8.9
Ubuntu-4.18.0-9.10
Ubuntu-raspi2-4.18.0-1004.4
Ubuntu-raspi2-4.18.0-1004.5
Ubuntu-raspi2-4.18.0-1004.6
I'm marking cosmic ["linux (Ubuntu)"] as Fix Released, too.

And just for the records, the kernel part already landed in xenial, too:
xenial updates kernel today: 4.4.0.137.
$ git log --oneline | grep "s390: detect etoken facility"
c32821c s390: detect etoken facility
$ git tag --contains c32821c
Ubuntu-4.4.0-136.162
Ubuntu-4.4.0-137.163
Ubuntu-4.4.0-138.164
Ubuntu-raspi2-4.4.0-1099.107
Ubuntu-snapdragon-4.4.0-1103.108

This bug was fixed in the package qemu - 1:2.11+dfsg-1ubuntu7.6

---------------
qemu (1:2.11+dfsg-1ubuntu7.6) bionic; urgency=medium

  [ Christian Ehrhardt ]
  * Add cpu model for z14 ZR1 (LP: #1780773)
  * d/p/ubuntu/lp-1789551-seccomp-set-the-seccomp-filter-to-all-threads.patch:
    ensure that the seccomp blacklist is applied to all threads (LP: #1789551)
    - CVE-2018-15746
  * improve s390x spectre mitigation with etoken facility (LP: #1790457)
    - debian/patches/ubuntu/lp-1790457-s390x-kvm-add-etoken-facility.patch
    - debian/patches/ubuntu/lp-1790457-partial-s390x-linux-headers-update.patch

  [ Phillip Susi ]
  * d/p/ubuntu/lp-1787267-fix-en_us-vnc-pipe.patch: Fix pipe, greater than and
    less than keys over vnc when using en_us kemaps (LP: #1787267).

 -- Christian Ehrhardt <email address hidden> Wed, 29 Aug 2018 11:46:37 +0200

The verification of the Stable Release Update for qemu has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

------- Comment From <email address hidden> 2018-10-19 04:59 EDT-------
IBM bugzilla status-> closed, Fix Released for all Distros in Service