Xen HVM guests running linux 4.10 fail to boot on Intel hosts

Bug #1671760 reported by Stefan Bader
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Won't Fix
Undecided
Unassigned
Trusty
Invalid
Undecided
Unassigned
Xenial
Invalid
Undecided
Unassigned
Yakkety
Invalid
Undecided
Unassigned
Zesty
Won't Fix
Undecided
Unassigned
xen (Ubuntu)
Fix Released
High
Unassigned
Trusty
Fix Released
High
Unassigned
Xenial
Fix Released
High
Unassigned
Yakkety
Fix Released
High
Unassigned
Zesty
Fix Released
High
Unassigned

Bug Description

Starting with Linux kernel 4.10, the kernel does some sanity checking on the TSC_ADJUST MSR. Xen has implemented some support for that MSR in the hypervisor (Xen 4.3 and later) for HVM guests. But boot and secondary vCPUs are set up inconsistently. This causes the boot of a 4.10 HVM guest to hang early on boot.

This was fixed in the hypervisor by:

  commit 98297f09bd07bb63407909aae1d309d8adeb572e
  x86/hvm: do not set msr_tsc_adjust on hvm_set_guest_tsc_fixed

That fix would be contained in 4.6.5 and 4.7.2 and would be in 4.8.1 (not released, yet) which mean that Ubuntu 14.04/16.04/16.10 and 17.04 currently are affected.

---

SRU Justification:

Impact: Without the TSC_ADJUST MSR fix, 4.10 and later kernels will get stuck at boot when running as (PV)HVM guests on Xen 4.3 and later.

Fix: Above fix either individually applied or as part of Xen stable stream (for Xen 4.7.x and 4.6.x) resolves the issue.

Testcase:
- Requires Intel based host which supports the TSC_ADJUST MSR
- Configured as Xen host
- HVM guest running Zesty/17.04
- Stuck at boot before, normal booting OS after

Revision history for this message
Stefan Bader (smb) wrote :
Changed in xen (Ubuntu Yakkety):
importance: Undecided → High
status: New → Triaged
Changed in xen (Ubuntu Xenial):
importance: Undecided → High
status: New → Triaged
Changed in xen (Ubuntu Trusty):
importance: Undecided → High
status: New → Triaged
tags: added: patch
Revision history for this message
Stefan Bader (smb) wrote :

Adding linux task to show the relationship between the two. Basically starting with 4.10 the kernel verifies the feature and exposes the broken implementation.

Changed in linux (Ubuntu Trusty):
status: New → Invalid
Changed in linux (Ubuntu Xenial):
status: New → Invalid
Changed in linux (Ubuntu Yakkety):
status: New → Invalid
Changed in linux (Ubuntu Zesty):
status: New → Won't Fix
Changed in linux (Ubuntu):
status: New → Won't Fix
tags: added: kernel-da-key
Revision history for this message
Stefan Bader (smb) wrote :

Verified against prepared releases in: https://launchpad.net/~smb/+archive/ubuntu/xen

Stefan Bader (smb)
description: updated
Stefan Bader (smb)
Changed in xen (Ubuntu Trusty):
status: Triaged → Fix Committed
Changed in xen (Ubuntu Xenial):
status: Triaged → Fix Committed
Changed in xen (Ubuntu Yakkety):
status: Triaged → Fix Committed
Changed in xen (Ubuntu Zesty):
status: Triaged → Fix Committed
assignee: Stefan Bader (smb) → nobody
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package xen - 4.8.0-1ubuntu2

---------------
xen (4.8.0-1ubuntu2) zesty; urgency=medium

  * Cherry-pick upstream change to fix TSC_ADJUST MSR handling in HVM
    guests running on Intel based hosts (LP: #1671760)

 -- Stefan Bader <email address hidden> Tue, 14 Mar 2017 09:27:04 +0100

Changed in xen (Ubuntu Zesty):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Stefan, or anyone else affected,

Accepted xen into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/xen/4.7.2-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-needed
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Stefan, or anyone else affected,

Accepted xen into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/xen/4.6.5-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Stefan, or anyone else affected,

Accepted xen into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/xen/4.4.2-0ubuntu0.14.04.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
Stefan Bader (smb) wrote :

For Xenial, reinstalled Xen from proposed. Booted Zesty 4.10 HVM ok.

tags: added: verification-done-xenial
Revision history for this message
Stefan Bader (smb) wrote :

For Yakkety, reinstalled Xen from proposed. Booted Zesty 4.10 HVM ok.

tags: added: verification-done-yakkety verification-needed-trusty
removed: verification-needed
Revision history for this message
Stefan Bader (smb) wrote :

For Xenial, reinstalled Xen from proposed. Booted Zesty 4.10 HVM ok.

Revision history for this message
Stefan Bader (smb) wrote :

... again. Had done that already last week...

Revision history for this message
Stefan Bader (smb) wrote :

Now successfully tested the Trusty proposed version.

tags: added: verification-done-trusty
removed: verification-needed-trusty
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package xen - 4.4.2-0ubuntu0.14.04.10

---------------
xen (4.4.2-0ubuntu0.14.04.10) trusty; urgency=medium

  * Backport upstream change to fix TSC_ADJUST MSR handling in HVM
    guests running on Intel based hosts (LP: #1671760)

 -- Stefan Bader <email address hidden> Tue, 14 Mar 2017 11:17:48 +0100

Changed in xen (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for xen has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (5.0 KiB)

This bug was fixed in the package xen - 4.6.5-0ubuntu1

---------------
xen (4.6.5-0ubuntu1) xenial; urgency=medium

  * Rebasing to upstream stable release 4.6.5 (LP: #1671864)
    https://www.xenproject.org/downloads/xen-archives/xen-46-series.html
    - Includes fix for booting 4.10 Linux kernels in HVM guests on Intel
      hosts which support the TSC_ADJUST MSR (LP: #1671760)
    - Additional security relevant changes:
      * CVE-2013-2076 / XSA-052 (update)
        - Information leak on XSAVE/XRSTOR capable AMD CPUs
      * CVE-2016-7093 / XSA-186 (4.6.3 became vulnerable)
        - x86: Mishandling of instruction pointer truncation during emulation
      * XSA-207
        - memory leak when destroying guest without PT devices
    - Replacing the following security fixes with the versions from the
      stable update:
      * CVE-2015-7812 / XSA-145
        - arm: Host crash when preempting a multicall
      * CVE-2015-7813 / XSA-146
        - arm: various unimplemented hypercalls log without rate limiting
      * CVE-2015-7814 / XSA-147
        - arm: Race between domain destruction and memory allocation decrease
      * CVE-2015-7835 / XSA-148
        - x86: Uncontrolled creation of large page mappings by PV guests
      * CVE-2015-7969 / XSA-149, XSA-151
        - leak of main per-domain vcpu pointer array
        - x86: leak of per-domain profiling-related vcpu pointer array
      * CVE-2015-7970 / XSA-150
        - x86: Long latency populate-on-demand operation is not preemptible
      * CVE-2015-7971 / XSA-152
        - x86: some pmu and profiling hypercalls log without rate limiting
      * CVE-2015-7972 / XSA-153
        - x86: populate-on-demand balloon size inaccuracy can crash guests
      * CVE-2016-2270 / XSA-154
        - x86: inconsistent cachability flags on guest mappings
      * CVE-2015-8550 / XSA-155
        - paravirtualized drivers incautious about shared memory contents
      * CVE-2015-5307, CVE-2015-8104 / XSA-156
        - x86: CPU lockup during exception delivery
      * CVE-2015-8338 / XSA-158
        - long running memory operations on ARM
      * CVE-2015-8339, CVE-2015-8340 / XSA-159
        XENMEM_exchange error handling issues
      * CVE-2015-8341 / XSA-160
        - libxl leak of pv kernel and initrd on error
      * CVE-2015-8555 / XSA-165
        - information leak in legacy x86 FPU/XMM initialization
      * XSA-166
        - ioreq handling possibly susceptible to multiple read issue
      * CVE-2016-1570 / XSA-167
        - PV superpage functionality missing sanity checks
      * CVE-2016-1571 / XSA-168
        - VMX: intercept issue with INVLPG on non-canonical address
      * CVE-2015-8615 / XSA-169
        - x86: unintentional logging upon guest changing callback method
      * CVE-2016-2271 / XSA-170
        - VMX: guest user mode may crash guest with non-canonical RIP
      * CVE-2016-3158, CVE-2016-3159 / XSA-172
        - broken AMD FPU FIP/FDP/FOP leak workaround
      * CVE-2016-3960 / XSA-173
        - x86 shadow pagetables: address width overflow
      * CVE-2016-4962 / XSA-175
        - Unsanitised guest input in libxl device handling code
      * CVE-2016-4480 / XSA-176
        - x86 ...

Read more...

Changed in xen (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package xen - 4.7.2-0ubuntu1

---------------
xen (4.7.2-0ubuntu1) yakkety; urgency=medium

  * Rebasing to upstream stable release 4.7.2 (LP: #1672767)
    https://www.xenproject.org/downloads/xen-archives/xen-47-series.html
    - Includes fix for booting 4.10 Linux kernels in HVM guests on Intel
      hosts which support the TSC_ADJUST MSR (LP: #1671760)
    - Dropping: d/p/preup-tools-fix-linear-p2m-save.patch which is part
      of the stable update.
    - Additional security relevant changes:
      * XSA-207
        - memory leak when destroying guest without PT devices
    - Replacing the following security fixes with the versions from the
      stable update:
      * CVE-2016-6258 / XSA-182
        - x86: Privilege escalation in PV guests
      * CVE-2016-6259 / XSA-183
        - x86: Missing SMAP whitelisting in 32-bit exception / event delivery
      * CVE-2016-7092 / XSA-185
        - x86: Disallow L3 recursive pagetable for 32-bit PV guests
      * CVE-2016-7093 / XSA-186
        - x86: Mishandling of instruction pointer truncation during emulation
      * CVE-2016-7094 / XSA-187
        - x86 HVM: Overflow of sh_ctxt->seg_reg[]
      * CVE-2016-7777 / XSA-190
        - CR0.TS and CR0.EM not always honored for x86 HVM guests
      * CVE-2016-9386 / XSA-191
        - x86 null segments not always treated as unusable
      * CVE-2016-9382 / XSA-192
        - x86 task switch to VM86 mode mis-handled
      * CVE-2016-9385 / XSA-193
        - x86 segment base write emulation lacking canonical address checks
      * CVE-2016-9384 / XSA-194
        - guest 32-bit ELF symbol table load leaking host data
      * CVE-2016-9383 / XSA-195
        - x86 64-bit bit test instruction emulation broken
      * CVE-2016-9377, CVE-2016-9378 / XSA-196
        - x86 software interrupt injection mis-handled
      * CVE-2016-9379, CVE-2016-9380 / XSA-198
        - delimiter injection vulnerabilities in pygrub
      * CVE-2016-9932 / XSA-200
        - x86 CMPXCHG8B emulation fails to ignore operand size override
      * CVE-2016-9815, CVE-2016-9816, CVE-2016-9817, CVE-2016-9818 / XSA-201
        - ARM guests may induce host asynchronous abort
      * CVE-2016-10024 / XSA-202
        - x86 PV guests may be able to mask interrupts
      * CVE-2016-10025 / XSA-203
        - x86: missing NULL pointer check in VMFUNC emulation
      * CVE-2016-10013 / XSA-204
        - x86: Mishandling of SYSCALL singlestep during emulation
  * Copy contents of debian/build/install-utils_$(ARCH)/usr/sbin into
    debian/build/install-utils_$ARCH/usr/lib/xen-$(VERSION) (LP: #1396670).

 -- Stefan Bader <email address hidden> Tue, 14 Mar 2017 15:45:59 +0100

Changed in xen (Ubuntu Yakkety):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.