Segmentation fault (kernel oops) with memory-hotplug in ubuntu_kernel_selftests on Bionic kernel

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1862312

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

This issue can be found on node modoc with Bionic as well
 [ 1148.018124] Injecting error (-12) to MEM_GOING_ONLINE
 [ 1148.019950] Injecting error (-12) to MEM_GOING_ONLINE
 [ 1148.021889] Injecting error (-12) to MEM_GOING_ONLINE
 [ 1149.494309] ------------[ cut here ]------------
 [ 1149.494314] kernel BUG at /build/linux-CWyQTi/linux-4.15.0/kernel/rcu/sync.c:128!
 [ 1149.494330] Oops: Exception in kernel mode, sig: 5 [#1]
 [ 1149.494333] LE SMP NR_CPUS=2048 NUMA PowerNV
 [ 1149.494338] Modules linked in: memory_notifier_error_inject notifier_error_inject overlay veth xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc iptable_filter binfmt_misc leds_powernv powernv_op_panel uio_pdrv_genirq uio ipmi_powernv ipmi_devintf powernv_rng ibmpowernv ipmi_msghandler vmx_crypto sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 ses enclosure scsi_transport_sas btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_vpmsum crc32c_vpmsum tg3 ipr [last unloaded: test_bpf]
 [ 1149.494411] CPU: 44 PID: 140088 Comm: mem-on-off-test Not tainted 4.15.0-87-generic #87-Ubuntu
 [ 1149.494416] NIP: c0000000001a8490 LR: c0000000001a8478 CTR: c00000000026c5e0
 [ 1149.494421] REGS: c000000f06d677c0 TRAP: 0700 Not tainted (4.15.0-87-generic)
 [ 1149.494425] MSR: 900000010282b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE,TM[E]> CR: 28222888 XER: 20000000
 [ 1149.494435] CFAR: c00000000001940c SOFTE: 1
 [ 1149.494435] GPR00: c0000000001a8434 c000000f06d67a40 c00000000172c900 0000000000000001
 [ 1149.494435] GPR04: 00000000000001f0 0000001ffdd10000 00000092c39e6cf7 0000000000000000
 [ 1149.494435] GPR08: 0000001ffdd10000 0000000000000001 0000000000000000 c000000f09be9390
 [ 1149.494435] GPR12: 0000000000002200 c00000000fa9e400 00000b644f99e4d0 0000000000000000
 [ 1149.494435] GPR16: 00000b6422f19690 00000b6422fa6ab0 00000b6422fd8204 00000b6422fa6ae8
 [ 1149.494435] GPR20: 00000b6422fdd5d8 0000000000000001 0000000000000000 00007fffd94fe1a4
 [ 1149.494435] GPR24: 00007fffd94fe1a0 c000000001763428 c0000000015f6ba8 0000000000000000
 [ 1149.494435] GPR28: 0000000000000020 c0000000015f6bb0 fffffffffffffff9 c0000000015f6ba8
 [ 1149.494482] NIP [c0000000001a8490] rcu_sync_enter+0xa0/0x1e0
 [ 1149.494487] LR [c0000000001a8478] rcu_sync_enter+0x88/0x1e0
 [ 1149.494490] Call Trace:
 [ 1149.494495] [c000000f06d67a40] [c000000f06d67aa0] 0xc000000f06d67aa0 (unreliable)
 [ 1149.494503] [c000000f06d67ab0] [c0000000001889a8] percpu_down_write+0x38/0x140
 [ 1149.494510] [c000000f06d67b00] [c00000000039fa6c] online_pages+0x1fc/0x440
 [ 1149.494517] [c000000f06d67bd0] [c0000000008a7320] memory_subsys_online+0x180/0x250
 [ 1149.494524] [c000000f06d67c60] [c000000000879f54] device_online+0x84/0x120
 [ 1149.494530] [c000000f06d67ca0] [c0000000008a7ee8] store_mem_state+0xb8/0x180
 [ 1149.494535] [c000000f06d67ce0] [c0000000008744bc] dev_attr_store+0x3c/0x60
 [ 1149.494541] [c000000f06d67d00] [c00...

This issue can be reproduced on X-4.15 Oracle (4.15.0-1032.35~16.04.1), instance BM.Standard1.36, VM.Standard1.4 (passed on VM.DenseIO1.8)

Issue found on 4.15.0-1059.61~16.04.1 AWS as well.
It's not failing on all of the instances.

A possible cause might be the Bionic commit 483735440add (mm/memory_hotplug: fix online/offline_pages called w.o. mem_hotplug_lock) that was backported from the upstream commit 381eab4a6ee8.

It seems the location of the new call to mem_hotplug_being() was shifted:

-->8--
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 6cd0c4a144d5..7dd8b84ea849 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -891,6 +891,8 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, int online_typ
  int ret;
  struct memory_notify arg;

+ mem_hotplug_begin();
+
  nid = pfn_to_nid(pfn);
  /* associate pfn range with the zone */
  zone = move_pfn_range(online_type, nid, pfn, nr_pages);
@@ -904,8 +906,6 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, int online_typ
  if (ret)
   goto failed_addition;

- mem_hotplug_begin();
-
  /*
   * If this zone is not populated, then it is not in zonelist.
   * This means the page allocator ignores this zone.
@@ -1178,9 +1178,7 @@ int __ref add_memory_resource(int nid, struct resource *res, bool online)
   walk_memory_range(PFN_DOWN(start), PFN_UP(start + size - 1),
       NULL, online_memory_block);

-
  return ret;
-
 error:
  /* rollback pgdat allocation and others */
  if (new_pgdat && pgdat)
-->8--

I will build a test kernel with this change to test it.

Hi, Sam.

Can you help me testing the change above? I uploaded a test kernel with this change to https://kernel.ubuntu.com/~mhcerri/lp1862312/

It's not clear to me what's the requirement to run the memory hotplug self test.

Thanks! I will try, but this issue was not spotted in our general kernel testing with the node we use (node rizzo). Thus it might take some time to find out an affected node first.

You can run this with the source tree:
$ make -C tools/testing/selftests TARGETS=memory-hotplug run_tests

Ref: https://www.kernel.org/doc/html/v4.15/dev-tools/kselftest.html

Issue NOT spotted on 4.15 GKE (4.15.0-1051.54), X-4.15 GCP (4.15.0-1053.57)

Spotted on B-4.15 AWS (4.15.0-1059.61) with instance r5.large

Hi Marcelo,
as it's much more easier to reproduce this on clouds (for now), so I just built a bionic AWS kernel with that commit reverted:
https://people.canonical.com/~phlin/kernel/lp-1862312-memhp/

The memory hotplug test works fine with multiple retries with this kernel. But if I switched to the 1059 in proposed, the r5.large instance will crash within 2 retries.

Fix for the issue: https://lists.ubuntu.com/archives/kernel-team/2020-February/107398.html

Tested on AWS with positive test results.

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

4.15.0-1060.62 linux-aws ubuntu_kernel_selftests has passed on all instances
4.15.0-1033.36 linux-oracle ubuntu_kernel_selftests has memory_hotplug test has passed
4.15.0-1052.55 linux-gke ubuntu_kernel_selftests has passed on all instances

awaiting main-kernel for amd64 and ppc64

Bionic 4.15 amd64, ppc64 are good.

Checking Xenial 4.15

Xenial 4.15, Xenial 4.15 Oracle all green.
thank you!

This bug was fixed in the package linux - 4.15.0-88.88

---------------
linux (4.15.0-88.88) bionic; urgency=medium

  * bionic/linux: 4.15.0-88.88 -proposed tracker (LP: #1862824)

  * Segmentation fault (kernel oops) with memory-hotplug in
    ubuntu_kernel_selftests on Bionic kernel (LP: #1862312)
    - Revert "mm/memory_hotplug: fix online/offline_pages called w.o.
      mem_hotplug_lock"
    - mm/memory_hotplug: fix online/offline_pages called w.o. mem_hotplug_lock

linux (4.15.0-87.87) bionic; urgency=medium

  * bionic/linux: 4.15.0-87.87 -proposed tracker (LP: #1861165)

  * Bionic update: upstream stable patchset 2020-01-22 (LP: #1860602)
    - scsi: lpfc: Fix discovery failures when target device connectivity bounces
    - scsi: mpt3sas: Fix clear pending bit in ioctl status
    - scsi: lpfc: Fix locking on mailbox command completion
    - Input: atmel_mxt_ts - disable IRQ across suspend
    - iommu/tegra-smmu: Fix page tables in > 4 GiB memory
    - scsi: target: compare full CHAP_A Algorithm strings
    - scsi: lpfc: Fix SLI3 hba in loop mode not discovering devices
    - scsi: csiostor: Don't enable IRQs too early
    - powerpc/pseries: Mark accumulate_stolen_time() as notrace
    - powerpc/pseries: Don't fail hash page table insert for bolted mapping
    - powerpc/tools: Don't quote $objdump in scripts
    - dma-debug: add a schedule point in debug_dma_dump_mappings()
    - clocksource/drivers/asm9260: Add a check for of_clk_get
    - powerpc/security/book3s64: Report L1TF status in sysfs
    - powerpc/book3s64/hash: Add cond_resched to avoid soft lockup warning
    - ext4: update direct I/O read lock pattern for IOCB_NOWAIT
    - jbd2: Fix statistics for the number of logged blocks
    - scsi: tracing: Fix handling of TRANSFER LENGTH == 0 for READ(6) and WRITE(6)
    - scsi: lpfc: Fix duplicate unreg_rpi error in port offline flow
    - f2fs: fix to update dir's i_pino during cross_rename
    - clk: qcom: Allow constant ratio freq tables for rcg
    - irqchip/irq-bcm7038-l1: Enable parent IRQ if necessary
    - irqchip: ingenic: Error out if IRQ domain creation failed
    - fs/quota: handle overflows of sysctl fs.quota.* and report as unsigned long
    - scsi: lpfc: fix: Coverity: lpfc_cmpl_els_rsp(): Null pointer dereferences
    - scsi: ufs: fix potential bug which ends in system hang
    - powerpc/pseries/cmm: Implement release() function for sysfs device
    - powerpc/security: Fix wrong message when RFI Flush is disable
    - scsi: atari_scsi: sun3_scsi: Set sg_tablesize to 1 instead of SG_NONE
    - clk: pxa: fix one of the pxa RTC clocks
    - bcache: at least try to shrink 1 node in bch_mca_scan()
    - HID: logitech-hidpp: Silence intermittent get_battery_capacity errors
    - libnvdimm/btt: fix variable 'rc' set but not used
    - HID: Improve Windows Precision Touchpad detection.
    - scsi: pm80xx: Fix for SATA device discovery
    - scsi: ufs: Fix error handing during hibern8 enter
    - scsi: scsi_debug: num_tgts must be >= 0
    - scsi: NCR5380: Add disconnect_mask module parameter
    - scsi: iscsi: Don't send data to unbound connection
    - scsi: target: iscsi: Wait for all commands to finish before freeing a
...