Botched backport breaks level triggered EOIs in QEMU guests with --machine kernel_irqchip=split

Bug #1644394 reported by Steve Rutherford
16
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
Medium
Luis Henriques
Xenial
Fix Released
Medium
Luis Henriques

Bug Description

The port of KVM's HyperV SynIC patches (and other HyperV related KVM Patches) broke QEMU's --machine kernel_irqchip=split for Ubuntu Xenial on x86. Guests hang in early boot.

Ubuntu Xenial dropped the upstream patch abdb080f7ac8a85547f5e0246362790043bbd3f2 while backporting. The result is that patch a6767645c9da8c0e91ebbb47677d933d6b378638 in Ubuntu's tree clobbers the definition of kvm_arch_irq_routing_update, which was renamed in abdb080f7..., and drops the IOAPIC scan request as a result. [Aside: this also results in kvm_hv_irq_routing_update being called outside irq_lock].

The result of this is that the EOI exit bitmaps are not updated when the table is updated, and no IOAPIC EOIs go to userspace.

The fix is backporting abdb080f7ac8a85547f5e0246362790043bbd3f2, and defining kvm_arch_post_irq_routing_update with the old definition of kvm_arch_port_irq_routing_update. I've attached a patch that does this.

Repro: The KVM-Unit-test's IOAPIC test should fail with QEMU and --machine kernel_irqchip=split on 4.4.0-36. (Caveat: I actually haven't tried this myself, but the theory is there.)

Security Implications: I believe this is not a security issue. kvm_hv_irq_routing_update only moved outside the lock, not outside the srcu critical section. If the update were outside the critical section, this would be a racy use-after-free.

Versioning information: I cloned the tree at git://kernel.ubuntu.com/ubuntu/ubuntu-xenial.git and checked out the tag 'Ubuntu-4.4.0-36.55'. All of the tests were done on a build of the 4.4.0-36 kernel. Later versions also have the issue (checked 'Ubuntu-4.4.0-49.70'). The issue was introduced in 'Ubuntu-4.4.0-32.51'.

Revision history for this message
Steve Rutherford (srutherford) wrote :
Revision history for this message
Steve Rutherford (srutherford) wrote :

[I've CC'ed the people that signed-off/acked the original backport.]

Revision history for this message
Brad Figg (brad-figg) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1644394

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Steve Rutherford (srutherford) wrote :

Those logs shouldn't be necessary. Identified the underlying bug by code inspection of your public tree (and local testing). Marking as confirmed.

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
tags: added: patch
Revision history for this message
Luis Henriques (henrix) wrote :

Submitted Steve's backport to the kernel team mailing-list for review:

https://lists.ubuntu.com/archives/kernel-team/2016-November/081152.html

Changed in linux (Ubuntu Xenial):
status: New → In Progress
assignee: nobody → Luis Henriques (henrix)
Luis Henriques (henrix)
Changed in linux (Ubuntu Xenial):
status: In Progress → Fix Committed
Revision history for this message
Steve Rutherford (srutherford) wrote :

Thanks!

Revision history for this message
Luis Henriques (henrix) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-xenial
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (17.0 KiB)

This bug was fixed in the package linux - 4.4.0-57.78

---------------
linux (4.4.0-57.78) xenial; urgency=low

  * Release Tracking Bug
    - LP: #1648867

  * Miscellaneous Ubuntu changes
    - SAUCE: Do not build the xr-usb-serial driver for s390

linux (4.4.0-56.77) xenial; urgency=low

  * Release Tracking Bug
    - LP: #1648867

  * Release Tracking Bug
    - LP: #1648579

  * CONFIG_NR_CPUS=256 is too low (LP: #1579205)
    - [Config] Increase the NR_CPUS to 512 for amd64 to support systems with a
      large number of cores.

  * NVMe drives in Amazon AWS instance fail to initialize (LP: #1648449)
    - SAUCE: (no-up) NVMe: only setup MSIX once

linux (4.4.0-55.76) xenial; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1648503

  * NVMe driver accidentally reverted to use GSI instead of MSIX (LP: #1647887)
    - (fix) NVMe: restore code to always use MSI/MSI-x interrupts

linux (4.4.0-54.75) xenial; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1648017

  * Update hio driver to 2.1.0.28 (LP: #1646643)
    - SAUCE: hio: update to Huawei ES3000_V2 (2.1.0.28)

  * linux: Enable live patching for all supported architectures (LP: #1633577)
    - [Config] CONFIG_LIVEPATCH=y for s390x

  * Botched backport breaks level triggered EOIs in QEMU guests with --machine
    kernel_irqchip=split (LP: #1644394)
    - kvm/irqchip: kvm_arch_irq_routing_update renaming split

  * Xenial update to v4.4.35 stable release (LP: #1645453)
    - x86/cpu/AMD: Fix cpu_llc_id for AMD Fam17h systems
    - KVM: x86: fix missed SRCU usage in kvm_lapic_set_vapic_addr
    - KVM: Disable irq while unregistering user notifier
    - fuse: fix fuse_write_end() if zero bytes were copied
    - mfd: intel-lpss: Do not put device in reset state on suspend
    - can: bcm: fix warning in bcm_connect/proc_register
    - i2c: mux: fix up dependencies
    - kbuild: add -fno-PIE
    - scripts/has-stack-protector: add -fno-PIE
    - x86/kexec: add -fno-PIE
    - kbuild: Steal gcc's pie from the very beginning
    - ext4: sanity check the block and cluster size at mount time
    - crypto: caam - do not register AES-XTS mode on LP units
    - drm/amdgpu: Attach exclusive fence to prime exported bo's. (v5)
    - clk: mmp: pxa910: fix return value check in pxa910_clk_init()
    - clk: mmp: pxa168: fix return value check in pxa168_clk_init()
    - clk: mmp: mmp2: fix return value check in mmp2_clk_init()
    - rtc: omap: Fix selecting external osc
    - iwlwifi: pcie: fix SPLC structure parsing
    - mfd: core: Fix device reference leak in mfd_clone_cell
    - uwb: fix device reference leaks
    - PM / sleep: fix device reference leak in test_suspend
    - PM / sleep: don't suspend parent when async child suspend_{noirq, late}
      fails
    - IB/mlx4: Check gid_index return value
    - IB/mlx4: Fix create CQ error flow
    - IB/mlx5: Use cache line size to select CQE stride
    - IB/mlx5: Fix fatal error dispatching
    - IB/core: Avoid unsigned int overflow in sg_alloc_table
    - IB/uverbs: Fix leak of XRC target QPs
    - IB/cm: Mark stale CM id's whenever the mad agent was unregistered
    - netfilter: nft_dynset: fix element timeou...

Changed in linux (Ubuntu Xenial):
status: Fix Committed → Fix Released
Changed in linux (Ubuntu Xenial):
importance: Undecided → Medium
Changed in linux (Ubuntu):
status: Confirmed → Fix Released
importance: Undecided → Medium
assignee: nobody → Luis Henriques (henrix)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.