Ubuntu 16.04.3 - call traces occurs when memory-hotplug test is run with 16Gb hugepages configured

------- Comment From <email address hidden> 2017-10-17 06:53 EDT-------
Patches submitted to mailing list.

Are we waiting on the patches referenced in the bug description and comment #1 to land upstream before backporting?

------- Comment From <email address hidden> 2017-11-30 11:36 EDT-------
@Praveen,

any updates on your testing for this on a PowerNV system?

hi Andrew,

I understand that the patches are already accepted on upstream release, aren't they?

I can see them on Linus' tree.

Hi Breno, thanks for the update. I'll move the bug to "Triage" and ask the kernel team to take a look.

An Artful SRU request for these commits was already submitted in bug 1706247. I'm working on backporting them to Xenial(4.4 based), but have had some issues with build failures. There are some prereq commits that are also needed for 4.4 kernels. I'll see what the extent of these prereqs are and post an update.

If there are too many prereqs for Xenial, we may have to have this issues address in Xenial via the HWE kernels, but I'm investigating.

I built a Zesty test kernel with the six requested patches.

The test kernel can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1706247/zesty

Can you test this kernel and see if it resolves this bug?

------- Comment From <email address hidden> 2017-12-19 05:45 EDT-------
Hi,

thank you for the patches. I'll test it.

------- Comment From <email address hidden> 2017-12-19 09:39 EDT-------
(In reply to comment #54)
> I built a Zesty test kernel with the six requested patches.
>
> The test kernel can be downloaded from:
>
> http://kernel.ubuntu.com/~jsalisbury/lp1706247/zesty
>
> Can you test this kernel and see if it resolves this bug?

Hi,

I wasn't able to find virtio.ko module in this package. Could you build an image with this module, please? Then, I can test this kernel on a LPAR machine as well.

Thank you

------- Comment From <email address hidden> 2017-12-19 14:01 EDT-------
The bug is not reproducible in PowerNV with the kernel provided in comment #54.

Waiting for a package with virtio module to run tests on a VM.

Did you install both the linux-image and linux-image-extra .deb packages? I believer the virtio module in in -extra.

------- Comment From <email address hidden> 2017-12-20 08:19 EDT-------
Hi,

I did some tests on a LPAR machine. I got the following:

[root@tuletapio1-lp1 memory-hotplug]# sh ./mem-on-off-test.sh -r 80
[...]
[ 204.004656] Offlined Pages 4096
[ 204.008378] Offlined Pages 4096
[ 204.012030] Offlined Pages 4096
[ 204.012201] Built 2 zonelists in Node order, mobility grouping on. Total pages: 92347
[ 204.012210] Policy zone: DMA
[ 204.012283] ------------[ cut here ]------------
[ 204.012286] kernel BUG at /home/jsalisbury/bugs/lp1706247/zesty/ubuntu-zesty/mm/slub.c:3993!
[ 204.012291] Oops: Exception in kernel mode, sig: 5 [#1]
[ 204.012294] SMP NR_CPUS=2048
[ 204.012295] NUMA
[ 204.012297] pSeries
[ 204.012300] Modules linked in: xt_tcpudp ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter btrfs xor raid6_pq vmx_crypto ip_tables x_tables xfs libcrc32c ibmveth ibmvscsi crc32c_vpmsum autofs4 [last unloaded: notifier_error_inject]
[ 204.012343] CPU: 18 PID: 6298 Comm: sh Not tainted 4.10.0-42-generic #46~lp1706247
[ 204.012347] task: c0000001f7dd8c00 task.stack: c0000001eef3c000
[ 204.012350] NIP: c000000000311134 LR: c000000000311148 CTR: c000000000310f40
[ 204.012353] REGS: c0000001eef3f6c0 TRAP: 0700 Not tainted (4.10.0-42-generic)
[ 204.012356] MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>
[ 204.012361] CR: 24222828 XER: 00000001
[ 204.012365] CFAR: c000000000311154 SOFTE: 1
GPR00: c000000000311148 c0000001eef3f940 c00000000145c900 0000000000000001
GPR04: c0000001fe010000 c0000001fe030000 c0000001fe030000 0000000000000000
GPR08: 0000000000000000 c0000001fe02fd88 0000000000000001 0000000000000000
GPR12: 0000000024222822 c00000000fb8a200 0000000000000bff f000000002ffffc0
GPR16: 0000000000000001 0000000000000000 c000000001483b00 0000000100001686
GPR20: 0000000000001000 0000000000000001 c0000001eef3fae0 c0000001ffec0600
GPR24: 0000000002fc0000 c0000000015ce368 f000000000000000 0000000000000000
GPR28: c000000001360738 0000000000000008 c000000001360758 c0000001fe02fd80
[ 204.012404] NIP [c000000000311134] slab_memory_callback+0x1f4/0x2b0
[ 204.012408] LR [c000000000311148] slab_memory_callback+0x208/0x2b0
[ 204.012410] Call Trace:
[ 204.012413] [c0000001eef3f940] [c000000000311148] slab_memory_callback+0x208/0x2b0 (unreliable)
[ 204.012419] [c0000001eef3f9a0] [c000000000112834] notifier_call_chain+0xa4/0x110
[ 204.012424] [c0000001eef3f9f0] [c000000000112d34] __blocking_notifier_call_chain+0x74/0xb0
[ 204.012429] [c0000001eef3fa40] [c000000000779a00] memory_notify+0x30/0x50
[ 204.012433] [c0000001eef3fa60] [c0000000003424cc] __offline_pages.constprop.8+0xa4c/0xa60
[ 204.012437] [c0000001eef3fbb0] [c000000000778afc] memory_block_action+0x9c/0x240
[ 204.012441] [c0000001eef3fc30] [c000000000779878] memory_subsys_offline+0x68/0xf0
[ 204...

Did the test kernel posted in comment #5 fix the original bug, but introduce a new issue?

------- Comment From <email address hidden> 2018-01-05 08:55 EDT-------
Laurent,

do you know maybe why this problem is occurring?

I get this problem always in the second run of the test script.

One think that is interesting is that if I reboot the system, it seems to work properly on the first run of the script, then, in the second run it fails.

------- Comment From <email address hidden> 2018-01-05 10:01 EDT-------
(In reply to comment #61)
> Laurent,
>
> do you know maybe why this problem is occurring?

I have no idea, but reading at the bug's history, it seems to be a new issue.

> I get this problem always in the second run of the test script.
>
> One think that is interesting is that if I reboot the system, it seems to
> work properly on the first run of the script, then, in the second run it
> fails.

Based on the stack trace, I would suggest to open a new bug for this issue.

------- Comment From <email address hidden> 2018-01-10 13:46 EDT-------
Hi Joseph,

so, it seems that we are dealing with a new bug!

Do you think it will be ok to open another bug with this kernel which you provided?

Thank you

Yes, it would be best to open a new bug. Just to confirm, this bug only happens when using the test kernel I posted in comment #5?

------- Comment From <email address hidden> 2018-01-16 10:45 EDT-------
Hi Joseph,

yes. We get a Segmentation Fault error without this patch an then I think we are unable to see that different error.

I'll open a new bug for it.

I removed the Zesty bug task since it is now EOL. Does this bug still happen with the latest HWE kernel for Xenial?

The latest HWE kernel should be 4.13 based.

------- Comment From <email address hidden> 2018-01-23 09:23 EDT-------
Hi Joseph,

I was able to reproduce the problem on HWE Kernal (XENIAL):

-- System

root@tuletapio2-lp3:~# uname -a
Linux tuletapio2-lp3 4.13.13 #1 SMP Tue Jan 23 07:41:39 CST 2018 ppc64le ppc64le ppc64le GNU/Linux

root@tuletapio2-lp3:~# cat /proc/meminfo
HugePages_Total: 2
HugePages_Free: 2
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 16777216 kB

root@tuletapio2-lp3:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinux-4.13.13 root=UUID=728bebfe-83ba-410d-917b-b552edbbb0a3 ro quiet splash default_hugepagesz=16G hugepagesz=16G hugepages=2

-- DUMP
Unable to handle kernel paging request for data at address 0x5deadbeef0000108
Faulting instruction address: 0xc0000000002cf374
Oops: Kernel access of bad area, sig: 11 [#1]
SMP NR_CPUS=2048
NUMA
pSeries
Modules linked in:
CPU: 7 PID: 3769 Comm: mem-on-off-test Not tainted 4.13.13 #1
task: c0000007d9400000 task.stack: c0000007d9480000
NIP: c0000000002cf374 LR: c0000000002cf298 CTR: c000000000134ef0
REGS: c0000007d94837d0 TRAP: 0380 Not tainted (4.13.13)
MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>
CR: 44422484 XER: 00000000
CFAR: c0000000002cf308 SOFTE: 1
GPR00: c0000000002cf28c c0000007d9483a50 c000000000e6ff00 c000000000fbc580
GPR04: f000000004000000 5deadbeef0000100 5deadbeef0000200 0000000000000002
GPR08: 5deadbeef0000000 0000000000000001 c000000000fbc580 c0000007fb000628
GPR12: 0000000000002200 c00000000fd02a00 0000000000001000 f000000004000000
GPR16: 0000000000000002 0000000000000000 c000000000ea3b00 0000000000001000
GPR20: 00000000ffffdb4f 0000000000000001 c000000fffd44600 c0000007d9483b60
GPR24: 0000000000100000 c000000000f0dad8 c000000000eb2dd0 0000000000000001
GPR28: 0000000000000000 c000000000fc4580 c000000000fd4580 f000000004000000
NIP [c0000000002cf374] dissolve_free_huge_page+0x124/0x230
LR [c0000000002cf298] dissolve_free_huge_page+0x48/0x230
Call Trace:
[c0000007d9483a50] [c0000000002cf28c] dissolve_free_huge_page+0x3c/0x230 (unreliable)
[c0000007d9483a90] [c0000000002cf548] dissolve_free_huge_pages+0xc8/0x150
[c0000007d9483ae0] [c0000000002ee1c8] __offline_pages.constprop.5+0x398/0xa90
[c0000007d9483c30] [c000000000645870] memory_subsys_offline+0x60/0xf0
[c0000007d9483c60] [c000000000623434] device_offline+0xf4/0x130
[c0000007d9483ca0] [c000000000645718] store_mem_state+0x178/0x190
[c0000007d9483ce0] [c00000000061ea34] dev_attr_store+0x34/0x60
[c0000007d9483d00] [c0000000003bdd10] sysfs_kf_write+0x60/0xa0
[c0000007d9483d20] [c0000000003bcaac] kernfs_fop_write+0x16c/0x240
[c0000007d9483d70] [c000000000314cf4] __vfs_write+0x34/0x70
[c0000007d9483d90] [c00000000031673c] vfs_write+0xcc/0x230
[c0000007d9483de0] [c000000000318410] SyS_write+0x60/0x110
[c0000007d9483e30] [c00000000000b860] system_call+0x58/0x6c
Instruction dump:
e90a0030 88ff0007 7fa94000 419e0120 3d005dea e8df0028 e8bf0020 7fe4fb78
6108dbee 7d435378 790807c6 6508f000 <f8c50008> f8a60000 7d094378 61080100
---[ end trace 23e3dda3fe0a58bd ]--

Can you see if this bug is resolved with the following artful kernel for bug 1706247:
http://kernel.ubuntu.com/~jsalisbury/lp1706247/artful

------- Comment From <email address hidden> 2018-01-24 07:29 EDT-------
Hi,

The first problem seems to be not occurring anymore (the segmentation fault problem).

Although we are still facing the second problem (in the trace below), we created a new bug report for it.

root@tuletapio2-lp3:~/linux/tools/testing/selftests/memory-hotplug# uname -a
Linux tuletapio2-lp3 4.13.0-16-generic #19~lp1706247 SMP Wed Nov 1 23:21:53 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux

[ 101.664761] kernel BUG at /home/jsalisbury/bugs/lp1706247/artful/ubuntu-artful/mm/slub.c:4034!
[ 101.664781] Oops: Exception in kernel mode, sig: 5 [#1]
[ 101.664785] SMP NR_CPUS=2048
[ 101.664785] NUMA
[ 101.664788] pSeries
[ 101.664792] Modules linked in: btrfs xor raid6_pq vmx_crypto crct10dif_vpmsum ip_tables x_tables autofs4 ibmveth ibmvscsi crc32c_vpmsum
[ 101.664807] CPU: 60 PID: 2151 Comm: sh Not tainted 4.13.0-16-generic #19~lp1706247
[ 101.664812] task: c000000fd2755900 task.stack: c000000fd27b8000
[ 101.664816] NIP: c0000000003630a4 LR: c0000000003630b8 CTR: c000000000150280
[ 101.664820] REGS: c000000fd27bb760 TRAP: 0700 Not tainted (4.13.0-16-generic)
[ 101.664824] MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>
[ 101.664829] CR: 24002828 XER: 00000000
[ 101.664835] CFAR: c0000000003630c4 SOFTE: 1
[ 101.664835] GPR00: c0000000003630b8 c000000fd27bb9e0 c000000001603300 0000000000000001
[ 101.664835] GPR04: c0000007fc010000 c0000007ffff3d80 f000000001ff03a0 0000000000000100
[ 101.664835] GPR08: c0000007ffff3c80 c0000007fc03fd90 0000000000000001 00000000ffffffff
[ 101.664835] GPR12: 0000000024002822 c00000000fae7600 f000000000000000 0000000000000000
[ 101.664835] GPR16: 00000000003fffff 0000000000140000 c0000000018e9400 f000000004ffffc0
[ 101.664835] GPR20: 0000000000000000 c000000001633b00 0000000000000002 00000000ffffb370
[ 101.664835] GPR24: 0000000000000001 0000000000001000 c000000fd27bbb70 0000000000000000
[ 101.664835] GPR28: c0000000014cadd8 0000000000000010 c0000000014cae38 c0000007fc03fd80
[ 101.664883] NIP [c0000000003630a4] slab_memory_callback+0x1d4/0x320
[ 101.664887] LR [c0000000003630b8] slab_memory_callback+0x1e8/0x320
[ 101.664890] Call Trace:
[ 101.664893] [c000000fd27bb9e0] [c0000000003630b8] slab_memory_callback+0x1e8/0x320 (unreliable)
[ 101.664901] [c000000fd27bba40] [c00000000012bacc] notifier_call_chain+0x9c/0x110
[ 101.664906] [c000000fd27bba90] [c00000000012c704] blocking_notifier_call_chain+0x64/0xa0
[ 101.664912] [c000000fd27bbad0] [c000000000824340] memory_notify+0x30/0x50
[ 101.664917] [c000000fd27bbaf0] [c000000000367678] __offline_pages.constprop.6+0xaa8/0xb10
[ 101.664923] [c000000fd27bbc30] [c0000000008241d8] memory_subsys_offline+0x68/0xd0
[ 101.664928] [c000000fd27bbc60] [c0000000007f7b68] device_offline+0xc8/0x140
[ 101.664933] [c000000fd27bbca0] [c000000000824040] store_mem_state+0x190/0x1b0
[ 101.664938] [c000000fd27bbce0] [c0000000007f27dc] dev_attr_store+0x3c/0x60
[ 101.664943] [c000000fd27bbd00] [c000000000469bd4] sysfs_kf_write+0x64/0x90
[ 101.664948] [c000000fd27bbd20] [c00000000046883c] kernfs_fop_write+0x1ac/0x270
[ 101.664954] [c000000fd27bbd70] [c00000000039d5cc] __vfs_write+0x3c...

Thanks for the update. I'll mark this bug as a duplicate of bug 1706247, since the fixes for that bug are the same for this bug.

We can look at the traces for the new bug in that bug report.

------- Comment From <email address hidden> 2018-07-18 09:56 EDT-------
Hi Joseph,

what is the status of this bug?

Tks

This bug has been marked as a duplicate of https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1706247. As a result, no updates will be posted to this bug. Please goto LP#1706247 for the current status.
Thanks.