Ubuntu
linux package

32-bit x86 kernel 4.15.0-50 crash in vmalloc_sync_all

Bionic (18.04)
Bug #1830433

Bug #1830433 reported by Mathieu Desnoyers on 2019-05-24

This bug affects 4 people

Affects		Status	Importance	Assigned to	Milestone
	linux (Ubuntu)	Fix Released	Medium	Unassigned
	Bionic	Fix Released	Medium	Unassigned

Bug Description

[Impact]

Commit d653420532d580156c8486686899ea6a9eeb7bf0 in bionic enabled kernel page table isolation for x86_32, but also introduced a kernel bug (the BUG_ON() condition in vmalloc_sync_one()) that seems to happen when vmalloc_sync_all() is called multiple times (e.g., in a busy loop).

The real problem seems to be a race condition with page-table entries' initialization that can be fixed applying the upstream commit 9bc4f28af75a91aea0ae383f50b0a430c4509303 ("x86/mm: Use WRITE_ONCE() when setting PTEs").

[Test Case]

The bug can be easily triggered by rebooting the system a couple of times and loading this module:

https://launchpadlibrarian.net/428142172/vmalloc-stress-test.c

[Fix]

The following upstream fix seems to resolve the problem:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9bc4f28af75a91aea0ae383f50b0a430c4509303

In addition to that the following other upstream fixes are required (all clean cherry picks) to do a cleaner backport of 9bc4f28af75a91aea0ae383f50b0a430c4509303:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=86fa949b050184ffc53688516a6a83ae5f98d08a
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=792adb90fa724ce07c0171cbc96b9215af4b1045
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5e0fb5df2ee871b841f96f9cb6a7f2784e96aa4e
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=785a19f9d1dd8a4ab2d0633be4656653bd3de1fc
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f967db0b9ed44ec3057a28f3b28efc51df51b835
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ba6f508d0ec4adb09f0a939af6d5e19cdfa8667d
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f77084d96355f5fba8e2c1fb3a51a393b1570de7

[Regression Potential]

All upstream fixes, tested on the affected platform, backport changes are minimal.

[Original bug report]

Hi,

I'm reproducing a kernel bug in vmalloc_sync_all() with a 32-bit x86 kernel.

The problem appears in

Linux ubuntu 4.15.0-50-generic #54-Ubuntu SMP Mon May 6 18:45:45 UTC 2019 i686 i686 i686 GNU/Linux

Kernels 4.15.0-49 and prior work fine.
The kernel 4.18.0-20-generic works fine.
This problem has not been experienced with upstream Linux kernels.

It appears that invoking vmalloc_sync_all() a few times end up triggering this issue. This can be triggered by restarting the lttng-sessiond service with lttng-modules-dkms installed (sometimes a few restarts are needed to trigger the bug). This ends up unloading and reloading those modules, which issues a few vmalloc_sync_all() as side-effect.

I'm not reporting this issue with the "ubuntu-bug linux" command because it crashes the system on that kernel (system hangs, no console output).

My test system runs within a kvm virtual machine on a 64-bit host.

lsb release:

Description: Ubuntu 18.04.2 LTS
Release: 18.04

Information about my kernel:

linux-image-4.15.0-50-generic:
  Installed: 4.15.0-50.54
  Candidate: 4.15.0-50.54
  Version table:
*** 4.15.0-50.54 500
        500 http://ca.archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages
        500 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages
        100 /var/lib/dpkg/status

Information about lttng-modules-dkms:

lttng-modules-dkms:
  Installed: 2.10.5-1ubuntu1.2
  Candidate: 2.10.5-1ubuntu1.2
  Version table:
*** 2.10.5-1ubuntu1.2 500
        500 http://ca.archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages
        100 /var/lib/dpkg/status
     2.10.5-1ubuntu1 500
        500 http://ca.archive.ubuntu.com/ubuntu bionic/universe i386 Packages

See original description

Tags:

CVE References

Revision history for this message

Mathieu Desnoyers (compudj) wrote on 2019-05-24:

kernel BUG screenshot Edit (36.9 KiB, image/png)

Revision history for this message

Mathieu Desnoyers (compudj) wrote on 2019-05-24:

version.log Edit (36 bytes, text/plain)

Revision history for this message

Mathieu Desnoyers (compudj) wrote on 2019-05-24:

uname-a.log Edit (99 bytes, text/plain)

Revision history for this message

Mathieu Desnoyers (compudj) wrote on 2019-05-24:

dmesg.log Edit (45.1 KiB, text/plain)

Revision history for this message

Mathieu Desnoyers (compudj) wrote on 2019-05-24:

lspci-vvnn.log Edit (8.5 KiB, text/plain)

Revision history for this message

Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote on 2019-05-24: Status changed to Confirmed

This change was made by a bot.

Changed in linux (Ubuntu):
status:	New → Confirmed
tags:	added: bionic

Revision history for this message

Michael Jeanson (mjeanson) wrote on 2019-05-24:

I built a minimal module that will trigger this bug when loaded/unloaded in a loop, usually happens in less than 30 seconds.

You can grab it here : https://github.com/mjeanson/boom

Build it with make and run the boom.sh script.

Andrea Righi (arighi) on 2019-06-13

Changed in linux (Ubuntu):
importance:	Undecided → Medium
Changed in linux (Ubuntu Bionic):
status:	New → Confirmed
importance:	Undecided → Medium

Revision history for this message

Andrea Righi (arighi) wrote on 2019-06-13:

vmalloc_sync_all() stress test Edit (734 bytes, text/x-csrc)

This module in attach (similar to the one posted by @mjeanson) has been used as an effective reproducer for this bug. It looks like we need to reboot the system a couple of times and load this module to immediately trigger the bug.

Andrea Righi (arighi) on 2019-06-13

description:

updated

Khaled El Mously (kmously) on 2019-07-01

Changed in linux (Ubuntu Bionic):
status:	Confirmed → Fix Committed

Revision history for this message

Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote on 2019-07-03:

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags:

added: verification-needed-bionic

Revision history for this message

NorthSec Infrastructure bot (northsec-bot) wrote on 2019-07-03:

#10

I was not able to trigger the crash on linux-image-4.15.0-55-generic=4.15.0-55.60 from -proposed. Looks like it's fixed.

Kleber Sacilotto de Souza (kleber-souza) on 2019-07-19

tags:

added: verification-done-bionic
removed: verification-needed-bionic

Revision history for this message

Launchpad Janitor (janitor) wrote on 2019-07-22:

#11

Download full text (11.2 KiB)

This bug was fixed in the package linux - 4.15.0-55.60

---------------
linux (4.15.0-55.60) bionic; urgency=medium

* linux: 4.15.0-55.60 -proposed tracker (LP: #1834954)

  * Request backport of ceph commits into bionic (LP: #1834235)
    - ceph: use atomic_t for ceph_inode_info::i_shared_gen
    - ceph: define argument structure for handle_cap_grant
    - ceph: flush pending works before shutdown super
    - ceph: send cap releases more aggressively
    - ceph: single workqueue for inode related works
    - ceph: avoid dereferencing invalid pointer during cached readdir
    - ceph: quota: add initial infrastructure to support cephfs quotas
    - ceph: quota: support for ceph.quota.max_files
    - ceph: quota: don't allow cross-quota renames
    - ceph: fix root quota realm check
    - ceph: quota: support for ceph.quota.max_bytes
    - ceph: quota: update MDS when max_bytes is approaching
    - ceph: quota: add counter for snaprealms with quota
    - ceph: avoid iput_final() while holding mutex or in dispatch thread

* QCA9377 isn't being recognized sometimes (LP: #1757218)
- SAUCE: USB: Disable USB2 LPM at shutdown

  * hns: fix ICMP6 neighbor solicitation messages discard problem (LP: #1833140)
    - net: hns: fix ICMP6 neighbor solicitation messages discard problem
    - net: hns: fix unsigned comparison to less than zero

* Fix occasional boot time crash in hns driver (LP: #1833138)
- net: hns: Fix probabilistic memory overwrite when HNS driver initialized

* use-after-free in hns_nic_net_xmit_hw (LP: #1833136)
- net: hns: fix KASAN: use-after-free in hns_nic_net_xmit_hw()

  * hns: attempt to restart autoneg when disabled should report error
    (LP: #1833147)
    - net: hns: Restart autoneg need return failed when autoneg off

  * systemd 237-3ubuntu10.14 ADT test failure on Bionic ppc64el (test-seccomp)
    (LP: #1821625)
    - powerpc: sys_pkey_alloc() and sys_pkey_free() system calls
    - powerpc: sys_pkey_mprotect() system call

  * [UBUNTU] pkey: Indicate old mkvp only if old and curr. mkvp are different
    (LP: #1832625)
    - pkey: Indicate old mkvp only if old and current mkvp are different

  * [UBUNTU] kernel: Fix gcm-aes-s390 wrong scatter-gather list processing
    (LP: #1832623)
    - s390/crypto: fix gcm-aes-s390 selftest failures

  * System crashes on hot adding a core with drmgr command (4.15.0-48-generic)
    (LP: #1833716)
    - powerpc/numa: improve control of topology updates
    - powerpc/numa: document topology_updates_enabled, disable by default

  * Kernel modules generated incorrectly when system is localized to a non-
    English language (LP: #1828084)
    - scripts: override locale from environment when running recordmcount.pl

  * [UBUNTU] kernel: Fix wrong dispatching for control domain CPRBs
    (LP: #1832624)
    - s390/zcrypt: Fix wrong dispatching for control domain CPRBs

  * CVE-2019-11815
    - net: rds: force to destroy connection if t_sock is NULL in
      rds_tcp_kill_sock().

  * Sound device not detected after resume from hibernate (LP: #1826868)
    - drm/i915: Force 2*96 MHz cdclk on glk/cnl when audio power is enabled
    - drm/i915: Save the old CDCLK atomic state
...

This bug was fixed in the package linux - 4.15.0-55.60

---------------
linux (4.15.0-55.60) bionic; urgency=medium

* linux: 4.15.0-55.60 -proposed tracker (LP: #1834954)

* QCA9377 isn't being recognized sometimes (LP: #1757218)
    - SAUCE: USB: Disable USB2 LPM at shutdown

* Fix occasional boot time crash in hns driver (LP: #1833138)
    - net: hns: Fix probabilistic memory overwrite when HNS driver initialized

*  use-after-free in hns_nic_net_xmit_hw (LP: #1833136)
    - net: hns: fix KASAN: use-after-free in hns_nic_net_xmit_hw()

* hns: attempt to restart autoneg when disabled should report error
    (LP: #1833147)
    - net: hns: Restart autoneg need return failed when autoneg off

* [UBUNTU] pkey: Indicate old mkvp only if old and curr. mkvp are different
    (LP: #1832625)
    - pkey: Indicate old mkvp only if old and current mkvp are different

* [UBUNTU] kernel: Fix gcm-aes-s390 wrong scatter-gather list processing
    (LP: #1832623)
    - s390/crypto: fix gcm-aes-s390 selftest failures

* Kernel modules generated incorrectly when system is localized to a non-
    English language (LP: #1828084)
    - scripts: override locale from environment when running recordmcount.pl

* [UBUNTU] kernel: Fix wrong dispatching for control domain CPRBs
    (LP: #1832624)
    - s390/zcrypt: Fix wrong dispatching for control domain CPRBs

* CVE-2019-11815
    - net: rds: force to destroy connection if t_sock is NULL in
      rds_tcp_kill_sock().

* Sound device not detected after resume from hibernate (LP: #1826868)
    - drm/i915: Force 2*96 MHz cdclk on glk/cnl when audio power is enabled
    - drm/i915: Save the old CDCLK atomic state
    - drm/i915: Remove redundant store of logical CDCLK state
    - drm/i915: Skip modeset for cdclk changes if possible

* Handle overflow in proc_get_long of sysctl (LP: #1833935)
    - sysctl: handle overflow in proc_get_long

* Dell XPS 13 (9370) defaults to s2idle sleep/suspend instead of deep, NVMe
    drains lots of power under s2idle (LP: #1808957)
    - Revert "UBUNTU: SAUCE: pci/nvme: prevent WDC PC SN720 NVMe from entering D3
      and being disabled"
    - Revert "UBUNTU: SAUCE: nvme: add quirk to not call disable function when
      suspending"
    - Revert "UBUNTU: SAUCE: pci: prevent Intel NVMe SSDPEKKF from entering D3"
    - Revert "SAUCE: nvme: add quirk to not call disable function when suspending"
    - Revert "SAUCE: pci: prevent sk hynix nvme from entering D3"
    - PCI: PM: Avoid possible suspend-to-idle issue
    - PCI: PM: Skip devices in D0 for suspend-to-idle
    - nvme-pci: Sync queues on reset
    - nvme: Export get and set features
    - nvme-pci: Use host managed power state for suspend

* linux v4.15 ftbfs on a newer host kernel (e.g. hwe) (LP: #1823429)
    - selinux: use kernel linux/socket.h for genheaders and mdp

* 32-bit x86 kernel 4.15.0-50 crash in vmalloc_sync_all (LP: #1830433)
    - x86/mm/pat: Disable preemption around __flush_tlb_all()
    - x86/mm: Drop usage of __flush_tlb_all() in kernel_physical_mapping_init()
    - x86/mm: Disable ioremap free page handling on x86-PAE
    - ioremap: Update pgtable free interfaces with addr
    - x86/mm: Add TLB purge to free pmd/pte page interfaces
    - x86/init: fix build with CONFIG_SWAP=n
    - x86/mm: provide pmdp_establish() helper
    - x86/mm: Use WRITE_ONCE() when setting PTEs

* hinic: fix oops due to race in set_rx_mode (LP: #1832048)
    - hinic: fix a bug in set rx mode

* ubuntu 18.04 flickering screen with Radeon X1600 (LP: #1791312)
    - drm/radeon: prefer lower reference dividers

* Login screen never appears on vmwgfx using bionic kernel 4.15 (LP: #1832138)
    - drm/vmwgfx: use monotonic event timestamps

* [linux-azure] Block Layer Commits Requested in Azure Kernels (LP: #1834499)
    - block: Clear kernel memory before copying to user
    - block/bio: Do not zero user pages

* CONFIG_LOG_BUF_SHIFT set to 14 is too low on arm64 (LP: #1824864)
    - [Config] CONFIG_LOG_BUF_SHIFT=18 on all 64bit arches

* Handle overflow for file-max (LP: #1834310)
    - sysctl: handle overflow for file-max
    - kernel/sysctl.c: fix out-of-bounds access when setting file-max

* [ALSA] [PATCH] Headset fixup for System76 Gazelle (gaze14) (LP: #1827555)
    - ALSA: hda/realtek - Headset fixup for System76 Gazelle (gaze14)
    - ALSA: hda/realtek - Corrected fixup for System76 Gazelle (gaze14)

* crashdump fails on HiSilicon D06 (LP: #1828868)
    - iommu/arm-smmu-v3: Abort all transactions if SMMU is enabled in kdump kernel
    - iommu/arm-smmu-v3: Don't disable SMMU in kdump kernel

* CVE-2019-11833
    - ext4: zero out the unused memory region in the extent tree block

* zfs 0.7.9 fixes a bug (https://github.com/zfsonlinux/zfs/pull/7343) that
    hangs the system completely (LP: #1772412)
    - SAUCE: (noup) Update zfs to 0.7.5-1ubuntu16.6

* does not detect headphone when there is no other output devices
    (LP: #1831065)
    - ALSA: hda/realtek - Fixed hp_pin no value
    - ALSA: hda/realtek - Use a common helper for hp pin reference

* kernel crash : net_sched  race condition in tcindex_destroy() (LP: #1825942)
    - net_sched: fix NULL pointer dereference when delete tcindex filter
    - RCU, workqueue: Implement rcu_work
    - net_sched: switch to rcu_work
    - net_sched: fix a race condition in tcindex_destroy()
    - net_sched: fix a memory leak in cls_tcindex
    - net_sched: initialize net pointer inside tcf_exts_init()
    - net_sched: fix two more memory leaks in cls_tcindex

* Support new ums-realtek device (LP: #1831840)
    - USB: usb-storage: Add new ID to ums-realtek

* amd_iommu possible data corruption (LP: #1823037)
    - iommu/amd: Reserve exclusion range in iova-domain
    - iommu/amd: Set exclusion range correctly

* Add new sound card PCIID into the alsa driver (LP: #1832299)
    - ALSA: hda: Add Icelake PCI ID
    - ALSA: hda/intel: add CometLake PCI IDs

* sky2 ethernet card doesn't work after returning from suspend
    (LP: #1807259) // sky2 ethernet card link not up after suspend
    (LP: #1809843)
    - sky2: Disable MSI on Dell Inspiron 1545 and Gateway P-79

* idle-page oopses when accessing page frames that are out of range
    (LP: #1833410)
    - mm/page_idle.c: fix oops because end_pfn is larger than max_pfn

* Add pointstick support on HP ZBook 17 G5 (LP: #1833387)
    - Revert "HID: multitouch: Support ALPS PTP stick with pid 0x120A"
    - SAUCE: HID: multitouch: Add pointstick support for ALPS Touchpad

* [SRU][B/B-OEM/B-OEM-OSP-1/C/D/E] Add trackpoint middle button support of 2
    new thinpads (LP: #1833637)
    - Input: elantech - enable middle button support on 2 ThinkPads

* CVE-2019-11085
    - drm/i915/gvt: Fix mmap range check
    - drm/i915: make mappable struct resource centric
    - drm/i915/gvt: Fix aperture read/write emulation when enable x-no-mmap=on

* CVE-2019-11884
    - Bluetooth: hidp: fix buffer overflow

* af_alg06 test from crypto test suite in LTP failed with kernel oops on B/C
    (LP: #1829725)
    - crypto: authenc - fix parsing key with misaligned rta_len

* CVE-2018-12126 // CVE-2018-12127 // CVE-2018-12130 // CVE-2019-11091
    - SAUCE: Synchronize MDS mitigations with upstream
    - Documentation: Correct the possible MDS sysfs values
    - x86/speculation/mds: Fix documentation typo

* CVE-2019-11091
    - x86/mds: Add MDSUM variant to the MDS documentation

* alignment test in powerpc from ubuntu_kernel_selftests failed on B/C Power9
    (LP: #1813118)
    - selftests/powerpc: Remove Power9 copy_unaligned test

* TRACE_syscall.ptrace_syscall_dropped in seccomp from ubuntu_kernel_selftests
    failed on B/C PowerPC (LP: #1812796)
    - selftests/seccomp: Enhance per-arch ptrace syscall skip tests

* Add powerpc/alignment_handler test for selftests (LP: #1828935)
    - selftests/powerpc: Add alignment handler selftest
    - selftests/powerpc: Fix to use ucontext_t instead of struct ucontext

* Cannot build kernel 4.15.0-48.51 due to an in-source-tree ZFS module.
    (LP: #1828763)
    - SAUCE: (noup) Update zfs to 0.7.5-1ubuntu16.5

* Eletrical noise occurred when external headset enter powersaving mode on a
    DEll machine (LP: #1828798)
    - ALSA: hda/realtek - Reduce click noise on Dell Precision 5820 headphone
    - ALSA: hda/realtek - Fixup headphone noise via runtime suspend

* [18.04/18.10] File libperf-jvmti.so is missing in linux-tools-common deb on
    Ubuntu (LP: #1761379)
    - [Packaging] Support building libperf-jvmti.so

* TCP : race condition on socket ownership in tcp_close() (LP: #1830813)
    - tcp: do not release socket ownership in tcp_close()

* bionic: netlink: potential shift overflow in netlink_bind() (LP: #1831103)
    - netlink: Don't shift on 64 for ngroups

* Add support to Comet Lake LPSS (LP: #1830175)
    - mfd: intel-lpss: Add Intel Comet Lake PCI IDs

* Reduce NAPI weight in hns driver from 256 to 64 (LP: #1830587)
    - net: hns: Use NAPI_POLL_WEIGHT for hns driver

* x86: add support for AMD Rome (LP: #1819485)
    - x86: irq_remapping: Move irq remapping mode enum
    - iommu/amd: Add support for higher 64-bit IOMMU Control Register
    - iommu/amd: Add support for IOMMU XT mode
    - hwmon/k10temp, x86/amd_nb: Consolidate shared device IDs
    - hwmon/k10temp: Add support for AMD family 17h, model 30h CPUs
    - x86/amd_nb: Add PCI device IDs for family 17h, model 30h
    - x86/MCE/AMD: Fix the thresholding machinery initialization order
    - x86/amd_nb: Add support for newer PCI topologies

* nx842 - CRB request time out (-110) when uninstall NX modules and initiate
    NX request (LP: #1827755)
    - crypto/nx: Initialize 842 high and normal RxFIFO control registers

* Require improved hypervisor detection patch in Ubuntu 18.04 (LP: #1829972)
    - s390/early: improve machine detection

-- Kleber Sacilotto de Souza <kleber.souza@canonical.com>  Tue, 02 Jul 2019 18:41:49 +0200

Changed in linux (Ubuntu Bionic):
status:	Fix Committed → Fix Released

Brad Figg (brad-figg) on 2019-07-24

tags:

added: cscc

Revision history for this message

Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote on 2019-08-22:

#12

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags:

added: verification-needed-xenial

Revision history for this message

Po-Hsu Lin (cypressyew) wrote on 2019-11-14:

#13

Patch available in Focal, mark this as fix-released.

Changed in linux (Ubuntu):
status:	Confirmed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

Duplicates of this bug

Bug #1828632

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntulinux package

32-bit x86 kernel 4.15.0-50 crash in vmalloc_sync_all

Bug Description

CVE References

Duplicates of this bug

Other bug subscribers

Bug attachments

Remote bug watches

Ubuntu
linux package