memory_stress_ng failing for Power architecture for 16.04

screen session of failure

kern.log from failure

screen session log from successful test on 14.04

kern.log from successful test on 14.04

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1573062

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

I don't see any evidence of a hang, just evidence of a machine being power-cycled.

Apr 19 21:26:33 ubuntu kernel: [19749.994340] [45742] 1000 45742 369 18 6 3 14 0 stress-ng-brk
Apr 19 21:26:33 ubuntu kernel: [19749.994342] [45743] 1000 45743 369 18 6 3 14 0 stress-ng-brk
Apr 19 21:26:33 ubuntu kernel: [19749.994344] Out of memory: Kill process 45583 (stress-ng-brk) score 28 or sacrifice child
Apr 19 21:26:33 ubuntu kernel: [19749.994566] Killed process 45583 (stress-ng-brk) total-vm:7976960kB, anon-rss:7048512kB, file-rss:1152kB
Apr 20 14:28:33 binacle kernel: [ 0.000000] opal: OPAL V3 detected !
Apr 20 14:28:33 binacle kernel: [ 0.000000] Allocated 4980736 bytes for 2048 pacas at c00000000fb40000
Apr 20 14:28:33 binacle kernel: [ 0.000000] Using PowerNV machine description
Apr 20 14:28:33 binacle kernel: [ 0.000000] Page sizes from device-tree:
Apr 20 14:28:33 binacle kernel: [ 0.000000] base_shift=12: shift=12, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=0
Apr 20 14:28:33 binacle kernel: [ 0.000000] base_shift=12: shift=16, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=7
Apr 20 14:28:33 binacle kernel: [ 0.000000] base_shift=12: shift=24, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=56
Apr 20 14:28:33 binacle kernel: [ 0.000000] base_shift=16: shift=16, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=1

Can you supply details of how stress-ng is being run?

For this brk stressor test, stress-ng performs rapid heap expansion using the brk() system call, which will force the system to consume all available memory and then transition into a swapping phase. This can lead to an apparent "hung" situation (for example, your shell may be swapped out), even though it is still active and not crashed.

The kernel messages you see in the kern.log are just the kernel OOM killer killing off the best candidate process to free up memory; this free'd memory will be consumed rapidly again by other contending brk stressors so the system will appear to be hung while it is busy cycling around on the killing and spawning of these stressors.

Was the machine ping'able? If so, it's not dead/hung. Perhaps it got power cycled prematurely. What evidence was there that the machine was hung?

The very first line of the screen session shows how the test is being run:

/usr/lib/plainbox-provider-checkbox/bin/memory_stress_ng

The machine is completely unresponsive. No ssh, no ping and the console over ipmi is locked up. Hitting CTRL+C over the sol console does nothing at all. This is after leaving the test running for 18+ hours where it should only take around 1-3 hours tops to complete depending on the machine. As you can see on the same machine running the same test on 14.04, the test took about an hour to complete with no lockup at the end.

Note that memory_stress_ng is a script in the Checkbox certification suite. The actual stress-ng command line should be:

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

The exact values of $end_time and $runtime vary with the amount of memory in the system -- it's 300 seconds plus 10 seconds per GiB for $runtime and 50% more for $end_time.

Note also that Mike's "1-3 hours tops" refers to the ENTIRE memory_stress_ng run; the "stress-ng... --brk 0" run would of course be much shorter than that, since the script runs a series of stressors in sequence.

Is this being run as a normal user or with root privileges?

It should be run as root.

Ahah, perhaps one should examine the stress-ng manual:

"Running stress-ng with root privileges will adjust out of memory settings on Linux systems to make the stressors unkillable in low memory situations, so use this judiciously. "

We may be just being a bit too demanding...

The tests run manually above were NOT run as root. The test during certification does run as root. I have run the test as root and as the ubuntu user, both with the same results.

Also, as shown above, this test ran fine manually while running Ubuntu 14.04 without root and during proper certification testing as root.

I'd like to perform a bisect to figure out what commit caused this regression. We need to identify the earliest kernel where the issue started happening as well as the latest kernel that did not have this issue.

Can you test the following kernels and report back? We are looking for the first kernel version that exhibits this bug:

3.16: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.16-utopic/
3.19: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19-vivid/
4.2: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.2-wily/

You don't have to test every kernel, just up until the kernel that first has this bug.

Once we know the results of those kernels, we can narrow it down further.

Thanks in advance!

Are there ppc64le versions of those kernels?

Mike - Wily was the first release to support ppc64el - http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.2-wily/linux-image-4.2.0-040200-generic_4.2.0-040200.201510260713_ppc64el.deb

Confirmed the above kernel fails in the same manner.

Can you test the 4.1 kernel next? It can be downloaded from:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1-wily/

We may also want to test the latest mainline kernel to see if this bug is already fixed there. If it is, we can perform a "Reverse" bisect to find the fix. The mainline kernel is available from:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.6-rc6-wily/

Ubuntu 14.04
3.19.0-58-generic - pass
4.2.0-35-generic - fails
4.1.0-040100-generic - fail
4.6.0-040600rc6-generic - pass

Ubuntu 16.04
4.4.0-22-generic - fail
4.1.0-040100-generic - fail
4.6.0-040600rc6-generic - pass

That is good news that the bug is fixed in mainline. We can perform a reverse bisect and find the commit that fixes this. We first need to narrow down which version fixes the issue. Can you test the following kernels next:

v4.5: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.5-wily/
v4.6-rc1: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.6-rc1-wily/

I'd like to chip in with my $0.02 worth. This may be a sporadic hang, so it may be worth double checking each bisection point in case it's a bug that may be a little more racy than we'd like.

I have noticed only on the Alpine hardware within an LPAR that is has been sporadic. I assume that is due to the additional hardware resources that can better handle the memory/stress test. All other tests have been 2 or 3 times each. More more than that. I can run the above kernel tests, but mind you, each test takes several hours to half a days worth. Running each test multiple times could take about a week to complete. We really need to get this sorted out ASAP since it is holding up all Power certification.

4.5.0-040500.201603140130_ppc64el - fail
4.6.0-040600rc1.201603261930_ppc64el - pass

4.6 was run twice and passed both times.

I started a "Reverse" kernel bisect between v4.5 final and v4.6-rc1. The kernel bisect will require testing of about 10-12 test kernels.

I built the first test kernel, up to the following commit:

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1573062

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

In parallel to the bisect, I'll review the git logs to see if the fix sticks out. There are 166 powerpc commits between these two versions, so a bisect may be the only way to identify the exact commit.

Thanks in advance

Testing now...

So far, after 2 runs, 4.5.0-040500-generic_4.5.0-040500.201605161244 seems to be working. I'm running it a 3rd time now just to be sure. It takes about 7 hours to finish.

Thanks for the update, Mike. If you think the two test runs are enough to prove that kernel is good, I'll build the next one. Otherwise, I'll wait for the results of the third run.

I would go ahead and assume it's good. I haven't had 2 tests run good twice in a row yet.

I built the next test kernel, up to the following commit:
53d2e6976bd4042672ed7b90dfbf4b31635b7dcf

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1573062

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

Commit 6b5f04b6cf8ebab9a65d9c0026c650bb2538fd0f was actually bad. So the test kernel in comment #28 is invalid. I'll re-update the bisect and build the next kernel.

I reset the bisect and built the first test kernel, up to the following commit:
96b9b1c95660d4bc5510c5d798d3817ae9f0b391

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1573062

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

Sorry we should actually test the kernel in comment #28. This is because we are performing a "Reverse" bisect and not a regular bisect. The test kernel is available from:

http://kernel.ubuntu.com/~jsalisbury/lp1573062

The .deb file name is:
linux-image-4.5.0-040500-generic_4.5.0-040500.201605171117_ppc64el.deb

Thanks, and sorry for the confusion.

4.5.0-040500.201605171117 failed

I built the next test kernel, up to the following commit:
2e11590171683c6b12193fe4b0ede1e6201b7f45

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1573062

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

PASS,2016-05-23-13-21-37,4.5.0-040500-generic-201605230752
FAIL,2016-05-23-21-06-07,4.5.0-040500-generic-201605230752
PASS,2016-05-24-04-32-39,4.5.0-040500-generic-201605230752
FAIL,2016-05-24-13-31-01,4.5.0-040500-generic-201605230752
FAIL,2016-05-25-04-06-51,4.5.0-040500-generic-201605230752

I built the next test kernel, up to the following commit:
5a010c73cdb760c9bdf37b28824b6566789cc005

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1573062

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

4.5.0-040500-generic-201605251558 failed

I built the next test kernel, up to the following commit:
8f40842e4260f73792c156aded004197a19135ee

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1573062

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

------- Comment From <email address hidden> 2016-05-27 03:40 EDT-------
Hi, where could I get the src/binary of memory_stress_ng. I will try to reproduce it in local power servers

Can you please post the bisection log using upstream commit ids.

8f40842e4260f73792c156aded004197a19135ee failed

Here is the current bisect log(This is a "Reverse" bisect so good means bad and vice versa):

git bisect start
# good: [b562e44f507e863c6792946e4e1b1449fbbac85d] Linux 4.5
git bisect good b562e44f507e863c6792946e4e1b1449fbbac85d
# bad: [f55532a0c0b8bb6148f4e07853b876ef73bc69ca] Linux 4.6-rc1
git bisect bad f55532a0c0b8bb6148f4e07853b876ef73bc69ca
# good: [6b5f04b6cf8ebab9a65d9c0026c650bb2538fd0f] Merge branch 'for-4.6' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup
git bisect good 6b5f04b6cf8ebab9a65d9c0026c650bb2538fd0f
# good: [53d2e6976bd4042672ed7b90dfbf4b31635b7dcf] Merge tag 'xfs-for-linus-4.6-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/dgc/linux-xfs
git bisect good 53d2e6976bd4042672ed7b90dfbf4b31635b7dcf
# good: [2e11590171683c6b12193fe4b0ede1e6201b7f45] staging: fsl-mc: fix incorrect type passed to dev_err macros
git bisect good 2e11590171683c6b12193fe4b0ede1e6201b7f45
# good: [5a010c73cdb760c9bdf37b28824b6566789cc005] Merge tag 'platform-drivers-x86-v4.6-1' of git://git.infradead.org/users/dvhart/linux-platform-drivers-x86
git bisect good 5a010c73cdb760c9bdf37b28824b6566789cc005
# good: [8f40842e4260f73792c156aded004197a19135ee] Merge tag 'for-linus-20160324' of git://git.infradead.org/linux-mtd
git bisect good 8f40842e4260f73792c156aded004197a19135ee

I built the next test kernel, up to the following commit:
45996492e5c85aa0ac93a95d1b2d1ed56851c865

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1573062

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

45996492e5c85aa0ac93a95d1b2d1ed56851c865 failed

I built the next test kernel, up to the following commit:
d9dddbf556674bf125ecd925b24e43a5cf2a568a

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1573062

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

d9dddbf556674bf125ecd925b24e43a5cf2a568a failed

I built the next test kernel, up to the following commit:
895a1067d5b83065afbad3bb02c3c464b71f1b3f

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1573062

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

895a1067d5b83065afbad3bb02c3c464b71f1b3f failed

Can you post the latest bisect log?

Commit failed:
89f081730c49a1d3b46359aa0054e6b3b80f47e4

Current "REVERSE" bisect log:

git bisect start
# good: [b562e44f507e863c6792946e4e1b1449fbbac85d] Linux 4.5
git bisect good b562e44f507e863c6792946e4e1b1449fbbac85d
# bad: [f55532a0c0b8bb6148f4e07853b876ef73bc69ca] Linux 4.6-rc1
git bisect bad f55532a0c0b8bb6148f4e07853b876ef73bc69ca
# good: [6b5f04b6cf8ebab9a65d9c0026c650bb2538fd0f] Merge branch 'for-4.6' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup
git bisect good 6b5f04b6cf8ebab9a65d9c0026c650bb2538fd0f
# good: [53d2e6976bd4042672ed7b90dfbf4b31635b7dcf] Merge tag 'xfs-for-linus-4.6-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/dgc/linux-xfs
git bisect good 53d2e6976bd4042672ed7b90dfbf4b31635b7dcf
# good: [2e11590171683c6b12193fe4b0ede1e6201b7f45] staging: fsl-mc: fix incorrect type passed to dev_err macros
git bisect good 2e11590171683c6b12193fe4b0ede1e6201b7f45
# good: [5a010c73cdb760c9bdf37b28824b6566789cc005] Merge tag 'platform-drivers-x86-v4.6-1' of git://git.infradead.org/users/dvhart/linux-platform-drivers-x86
git bisect good 5a010c73cdb760c9bdf37b28824b6566789cc005
# good: [8f40842e4260f73792c156aded004197a19135ee] Merge tag 'for-linus-20160324' of git://git.infradead.org/linux-mtd
git bisect good 8f40842e4260f73792c156aded004197a19135ee
# good: [45996492e5c85aa0ac93a95d1b2d1ed56851c865] orangefs: fix orangefs_superblock locking
git bisect good 45996492e5c85aa0ac93a95d1b2d1ed56851c865
# good: [d9dddbf556674bf125ecd925b24e43a5cf2a568a] mm/page_alloc: prevent merging between isolated and other pageblocks
git bisect good d9dddbf556674bf125ecd925b24e43a5cf2a568a
# good: [895a1067d5b83065afbad3bb02c3c464b71f1b3f] Merge tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi
git bisect good 895a1067d5b83065afbad3bb02c3c464b71f1b3f
# good: [89f081730c49a1d3b46359aa0054e6b3b80f47e4] libceph: use sizeof_footer() more
git bisect good 89f081730c49a1d3b46359aa0054e6b3b80f47e4

Thanks.

I see very little left between good and bad which could explain the bug going away. So I'm wondering if the bisect has gone off the rails.

I built the next test kernel, up to the following commit:
5ee61e95b6b33c82f6fa1382585faed66aa01245

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1573062

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

Can we get answer to #38. If we have the test that caused the failure we will try to reproduce here.

The memory_stress_ng test is part of the plainbox-provider-checkbox package which can be installed from the hardware-certification PPA:

https://code.launchpad.net/~hardware-certification/+archive/ubuntu/public

The full path once installed is:
/usr/lib/plainbox-provider-checkbox/bin/memory_stress_ng

Commit failed:
ee61e95b6b33c82f6fa1382585faed66aa01245

I built the next test kernel, up to the following commit:
ddc8f6feec76b5deea8090db015920a283006044

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1573062

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

Commit failed:
ddc8f6feec76b5deea8090db015920a283006044

I built the next test kernel, up to the following commit:
4f1b50c3e3082b31c94cee2b897bd9f5d0f3e7c8

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1573062

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

Commit failed:
4f1b50c3e3082b31c94cee2b897bd9f5d0f3e7c8

I built the next test kernel, up to the following commit:
698f415cf5756e320623bdb015a600945743377c

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1573062

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

I have 14.04 installed with 4.4.0-28 and I can see the following

In the bad case

1. OOM'ing of stress-ng-brk is slow, I can see it making progress -- see tasks being scheduled/console output and sysrq output on Ctrl-o h
2. stress-ng-brk is trying to make progress in OOM, but is heavily contenting for what seems like lru lock
3. The network driver is trying to serve softirq's and fails to process them as allocation fails and does a dump_stack()

In the other case (based on logs and limited testing - kernel 698f415cf5756e320623bdb015a600945743377c for me)
1. OOM proceeds quickly and stress-ng-brk is OOM'd frequently
2. At some point when all stress-ng-brk seem to be OOM'd the test completes

I think the test relies on all stress-ng-brk's to OOM before considering completion (I could be wrong), but in this case progress is slow, but the system is responding. I am going to try the kernel listed here

We believe that the test may require multiple iterations of running for check for consistency of commits.

At this point I think the test will progress and needs longer and progress is slow. I think the expectation is that it needs to complete faster; the system is not lock'd up completely as far as I can tell.

Sorry the 14.04 should be 16.04 in comment #61

------- Comment From <email address hidden> 2016-07-01 02:36 EDT-------
What is the criteria for forward progress of the stress?

I did a quick check for what processes are OOM'd

In new kernel

2 apport
1 cron
1 dhclient
1 gmain
2 in:imklog
1 (journald)
1 kworker/u160:4
1 rs:main
1 stress-ng
972 stress-ng-bighe
157 stress-ng-brk
10 swapper/1
2 swapper/16
17 swapper/2
39 swapper/40
15 swapper/41
1 swapper/65
3 systemd
3 systemd-cgroups
10 systemd-journal
1 systemd-logind

In the 14.04 kernel

1 dhclient
1 in:imklog
1 in:imuxsock
3 irqbalance
1 jbd2/sda2-8
1 kworker/u160:1
1 stress-ng
226 stress-ng-brk
32 swapper/16
3 swapper/23
2 swapper/31
3 swapper/32
22 swapper/46
5 swapper/55
33 swapper/56
3 swapper/63
6 swapper/64
18 swapper/7
6 swapper/72
7 swapper/8

We changed the OOM killer in 4.6 (see https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=aac453635549699c13a84ea1456d5b0e574ef855). Looks like we have good behaviour with 4.6 which could be a result of the change. I am yet to look at the source of memstress_ng, but if the
processes selected for OOM impact the result of the test, we could have a probable explanation. It will also be interesting to continue the bisect and see where we end up.

------- Comment From <email address hidden> 2016-07-06 20:56 EDT-------
From what I can see the following is the root cause of the issue

cgroup_threadgroup_rwsem almost serializes accesses on the system

1. stress-ng-brk has cgroup_threadgroup_rwsem held in read mode via copy_process() and does a schedule_timeout() from __alloc_pages_nodemask() which never seems to return from schedule_timeout()

00003fff96ceeab0 0 2799 2701 0x00040002
[ 4401.831972] Call Trace:
[ 4401.831973] [c000000ee71433c0] [c000000ee7143400] 0xc000000ee7143400 (unreliable)
[ 4401.831975] [c000000ee7143590] [c000000000017c64] __switch_to+0x204/0x360
[ 4401.831977] [c000000ee71435e0] [c000000000bb917c] __schedule+0x40c/0xe70
[ 4401.831979] [c000000ee71436a0] [c000000000bb9c34] schedule+0x54/0xd0
[ 4401.831981] [c000000ee71436d0] [c000000000bc0524] schedule_timeout+0x384/0x4f0
[ 4401.831983] [c000000ee7143800] [c00000000027de1c] __alloc_pages_nodemask+0xd0c/0xf40
[ 4401.831985] [c000000ee7143a10] [c0000000002e8d40] alloc_pages_current+0xc0/0x240
[ 4401.831988] [c000000ee7143a70] [c000000000056b6c] page_table_alloc+0xcc/0x1e0
[ 4401.831989] [c000000ee7143ac0] [c0000000002b5824] __pte_alloc+0x54/0x1e0
[ 4401.831991] [c000000ee7143b10] [c0000000002b8584] copy_page_range+0x754/0x8f0
[ 4401.831993] [c000000ee7143c40] [c0000000000bcee4] copy_process.isra.6+0x1834/0x1ab0
[ 4401.831995] [c000000ee7143d60] [c0000000000bd33c] _do_fork+0xac/0x980
[ 4401.831997] [c000000ee7143e30] [c00000000000946c] ppc_clone+0x8/0xc

[ 4401.861569] cfs_rq[23]:/user.slice
[ 4401.861570] .exec_clock : 1725230.642232
[ 4401.861571] .MIN_vruntime : 0.000001
[ 4401.861572] .min_vruntime : 1154678.434341
[ 4401.861573] .max_vruntime : 0.000001
[ 4401.861573] .spread : 0.000000
[ 4401.861574] .spread0 : -97866589.605918
[ 4401.861575] .nr_spread_over : 11
[ 4401.861575] .nr_running : 0

[ 4401.862187] stress-ng-brk 2799 1154678.007061 854611 120 688670.967816 1148995.803148 2318289.407734 0 0 /user.slice

2. Since cgroup_threadgroup_rwsem is grabbed, we are unable to make any processes exit

[ 4177.396262] Showing all locks held in the system:
[ 4177.396263] 4 locks held by systemd/1:
[ 4177.396268] #0: (sb_writers#9){.+.+.+}, at: [<c000000000328810>] __sb_start_write+0x100/0x130
[ 4177.396272] #1: (&of->mutex){+.+.+.}, at: [<c0000000003e76ec>] kernfs_fop_write+0x7c/0x1f0
[ 4177.396275] #2: (cgroup_mutex){+.+.+.}, at: [<c0000000001b0b8c>] cgroup_kn_lock_live+0x14c/0x280
[ 4177.396278] #3: (&cgroup_threadgroup_rwsem){++++++}, at: [<c00000000013aa10>] percpu_down_write+0x50/0x180

I think at #3, we are waiting for all readers to exit cgroup_threadgroup_rwsem, this further blocks exiting threads

[ 4177.396548] #0: (&cgroup_threadgroup_rwsem){++++++}, at: [<c0000000000d9b00>] exit_signals+0x50/0x1a0
[ 4177.396548] 1 lock held by kworker/dying/1348:
[ 4177.396551] #0: (&cgroup_threadgroup_rwsem){++++++}, at: [<c0000000000d9b00>] exit_signals+0x50/0x1a0
[ 4177.396552] 1 lock held by kworker/dying/1919:
[ 4177.396555...

I built a Xenial test kernel[0] with the fix[1] suggested in comment #64. The fix was mentioned upstream[2].

That particular patch has not landed in mainline yet, but it was also cc'd to stable for all 4.4+ kernels.

This commit can be added as a SAUCE patch if we find that it resolves this bug, it will be SAUCE until the real patch is applied through the normal stable update process.

Can you test this kernel and see if it resolves this bug? Note, with this test kernel you need to install both the linux-image and linux-image-extra .deb packages.

[0] http://kernel.ubuntu.com/~jsalisbury/lp1573062/
[1] https://patchwork.kernel.org/patch/8902401/
[2] https://lkml.org/lkml/2016/4/17/56

------- Comment From <email address hidden> 2016-07-10 23:00 EDT-------
No luck with the new build shared (just 1 run, I'll try more runs).. More debugging in progress as well

------- Comment From <email address hidden> 2016-07-11 00:59 EDT-------
Can you please provide links to the sources as well, just to do a quick diff against the 4.5 working git?

Have we made further progress on bisect?

Default Comment by Bridge

kern.log attached to show additional failures with patched kernel

------- Comment From <email address hidden> 2016-07-12 04:14 EDT-------
I backported the oom-reaper changes from v4.5 and I've had good runs so far (2 runs with machine returning to console)

I took aac453635549699c13a84ea1456d5b0e574ef855 + next 7 patches and removed unsupported bits. I also took the changes for schedule_timeout_idle() + memcontrol changes I pointed out earlier.

Hi,

Joe is away this week and we'd like to keep this bug moving forward. Based on comment #70, we'd like to build a new test kernel to try internally. However, we want to ensure we've applied the exact same changes/patches as noted in comment #70. Could we be pointed at a git repo with the patches? Or could they be attached here to the bug? Thanks in advance.

------- Comment on attachment From <email address hidden> 2016-07-13 07:53 EDT-------

These are fixes from 4.5 mostly OOM reaper + changes for compilation during the backport

It also includes schedule_timeout_idle() and the memcontrol.c fixes

Xenial (4.4.0-32.51~lp1573062) ppc64el test kernel comprised of 4.4.0-31.50 plus the changes from balbirs' comment #72's oom_reaper_4.4.diff*:

    http://people.canonical.com/~kamal/lp1573062/

*The source branch for that test kernel, with the original mainline commits (and LKML patch) split back out (resulting in a delta identical to comment #72 oom_reaper_4.4.diff):

    https://git.launchpad.net/~kamalmostafa/ubuntu/+source/linux/+git/xenial/log/?h=lp1573062

------- Comment From <email address hidden> 2016-07-13 23:04 EDT-------
I also added af8e15cc85a253155fdcea707588bf6ddfc0be2e to my diff, just FYI

Per comment #74, I've now also added af8e15cc85a253155fdcea707588bf6ddfc0be2e ("oom, oom_reaper: do not enqueue task if it is on the oom_reaper_list head") to the test kernel and source branch.

Xenial (4.4.0-32.51~lp1573062.1) ppc64el test kernel:

  http://people.canonical.com/~kamal/lp1573062/lp1573062.1/

  https://git.launchpad.net/~kamalmostafa/ubuntu/+source/linux/+git/xenial/log/?h=lp1573062

4.4.0-32-generic-51~lp1573062.1 failed.

------- Comment From <email address hidden> 2016-07-17 21:43 EDT-------
I tried the kernel at http://people.canonical.com/~kamal/lp1573062/lp1573062.1/ and it worked fine for me

It looks like there are conflicting test results with the 4.4.0-32-generic-51~lp1573062.1 test kernel.

@balbirs, can you confirm the test was run enough times to reproduce the bug?

@Mike Rushton, can you confirm that you can make the test fail multiple times with that kernel?

I'm running the tests as we speak across 2 LPAR's and 2 openpower PowerNV installations. One of the open power boxes has 256G of memory so it takes up to 7 hours for a test to pass or to be determined failed.

Attached is the kern.log from the Firestone PowerNV server. This one is failing with stack traces which only started after we started testing patched kernels.

The Habanero PowerNV failed with the typical locked up console and OOM's.

------- Comment From <email address hidden> 2016-07-17 21:43 EDT-------
I tried the kernel at http://people.canonical.com/~kamal/lp1573062/lp1573062.1/ and it worked fine for me

------- Comment From <email address hidden> 2016-07-19 01:04 EDT-------
Looks like I got a failure with the run on http://people.canonical.com/~kamal/lp1573062/lp1573062.1/

But with my diff + 4.4.0 source from apt-source, I can always get the the following command to succeed.

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

I've tried three times with my diff (all success) and twice with the kernel @ ~kamal (one failure and one success). I've not tried the longer 7 hour run

------- Comment From <email address hidden> 2016-07-19 01:37 EDT-------
In the kern.log posted, it looks like the problem has moved to

rwsem_wake+0xcc/0x110
up_write+0x78/0x90
unlink_anon_vmas+0x15c/0x2c0

A bunch of threads are stuck on rwsem_wake -- spinning on the sem->wait_lock. I can see a whole bunch of exiting stress-ng-mmapf stuck on this lock, spinning. I'll double check this. Can we get a build with lockdep enabled? I am unable to reproduce this issue at my end with the diff applied on my machine at the moment

@balbirs, would it be possible for you to send the individual .patch files, so I can apply them with "git am"? That will confirm we are using the same code base.

You can generate them with something like:

'git format-patch -k HEAD~10'

Change the number '10' to the number of commits you added with your changes/diffs. Alternatively, if you have your repo somewhere I can clone from, I can do that.

I can then build a test kernel with your patches, and build a test kernel with lockdep enabled.

------- Comment From <email address hidden> 2016-07-17 21:43 EDT-------
I tried the kernel at http://people.canonical.com/~kamal/lp1573062/lp1573062.1/ and it worked fine for me

------- Comment From <email address hidden> 2016-07-19 01:04 EDT-------
Looks like I got a failure with the run on http://people.canonical.com/~kamal/lp1573062/lp1573062.1/

But with my diff + 4.4.0 source from apt-source, I can always get the the following command to succeed.

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

I've tried three times with my diff (all success) and twice with the kernel @ ~kamal (one failure and one success). I've not tried the longer 7 hour run

------- Comment From <email address hidden> 2016-07-19 01:37 EDT-------
In the kern.log posted, it looks like the problem has moved to

rwsem_wake+0xcc/0x110
up_write+0x78/0x90
unlink_anon_vmas+0x15c/0x2c0

A bunch of threads are stuck on rwsem_wake -- spinning on the sem->wait_lock. I can see a whole bunch of exiting stress-ng-mmapf stuck on this lock, spinning. I'll double check this. Can we get a build with lockdep enabled? I am unable to reproduce this issue at my end with the diff applied on my machine at the moment

------- Comment From <email address hidden> 2016-07-19 19:51 EDT-------
I am cloning the sources to debug further

------- Comment From <email address hidden> 2016-07-17 21:43 EDT-------
I tried the kernel at http://people.canonical.com/~kamal/lp1573062/lp1573062.1/ and it worked fine for me

------- Comment From <email address hidden> 2016-07-19 01:04 EDT-------
Looks like I got a failure with the run on http://people.canonical.com/~kamal/lp1573062/lp1573062.1/

But with my diff + 4.4.0 source from apt-source, I can always get the the following command to succeed.

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

I've tried three times with my diff (all success) and twice with the kernel @ ~kamal (one failure and one success). I've not tried the longer 7 hour run

------- Comment From <email address hidden> 2016-07-19 01:37 EDT-------
In the kern.log posted, it looks like the problem has moved to

rwsem_wake+0xcc/0x110
up_write+0x78/0x90
unlink_anon_vmas+0x15c/0x2c0

A bunch of threads are stuck on rwsem_wake -- spinning on the sem->wait_lock. I can see a whole bunch of exiting stress-ng-mmapf stuck on this lock, spinning. I'll double check this. Can we get a build with lockdep enabled? I am unable to reproduce this issue at my end with the diff applied on my machine at the moment

------- Comment From <email address hidden> 2016-07-19 19:51 EDT-------
I am cloning the sources to debug further

------- Comment From <email address hidden> 2016-07-19 23:52 EDT-------
I cloned the kernel from https://git.launchpad.net/~kamalmostafa/ubuntu/+source/linux/+git/xenial/log/?h=lp1573062 and built with the machine config specified from /boot/config. I also verified the diff matches my changes.

I ran

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

twice

Both the times, the test did the right thing. Could someone verify if

(a) The smaller subset works fine?
(b) The larger test fails, if so, can we get a run with lockdep

I was just testing for the command line above and I could see a difference with those patches.

@balbirs, are you saying the repo you cloned from https://git.launchpad.net/~kamalmostafa/ubuntu/+source/linux/+git/xenial/log/?h=lp1573062 differs from your diffs?

------- Comment From <email address hidden> 2016-07-17 21:43 EDT-------
I tried the kernel at http://people.canonical.com/~kamal/lp1573062/lp1573062.1/ and it worked fine for me

------- Comment From <email address hidden> 2016-07-19 01:04 EDT-------
Looks like I got a failure with the run on http://people.canonical.com/~kamal/lp1573062/lp1573062.1/

But with my diff + 4.4.0 source from apt-source, I can always get the the following command to succeed.

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

I've tried three times with my diff (all success) and twice with the kernel @ ~kamal (one failure and one success). I've not tried the longer 7 hour run

------- Comment From <email address hidden> 2016-07-19 01:37 EDT-------
In the kern.log posted, it looks like the problem has moved to

rwsem_wake+0xcc/0x110
up_write+0x78/0x90
unlink_anon_vmas+0x15c/0x2c0

A bunch of threads are stuck on rwsem_wake -- spinning on the sem->wait_lock. I can see a whole bunch of exiting stress-ng-mmapf stuck on this lock, spinning. I'll double check this. Can we get a build with lockdep enabled? I am unable to reproduce this issue at my end with the diff applied on my machine at the moment

------- Comment From <email address hidden> 2016-07-19 19:51 EDT-------
I am cloning the sources to debug further

------- Comment From <email address hidden> 2016-07-19 23:52 EDT-------
I cloned the kernel from https://git.launchpad.net/~kamalmostafa/ubuntu/+source/linux/+git/xenial/log/?h=lp1573062 and built with the machine config specified from /boot/config. I also verified the diff matches my changes.

I ran

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

twice

Both the times, the test did the right thing. Could someone verify if

(a) The smaller subset works fine?
(b) The larger test fails, if so, can we get a run with lockdep

I was just testing for the command line above and I could see a difference with those patches.

------- Comment From <email address hidden> 2016-07-21 20:14 EDT-------
No, the diff matches, sorry for the confusion, but here is what I said

"I also verified the diff matches my changes"

In summary, here is what I did

1. cloned the sources
2. built locally on my machine
3. Ran stress-ng with recommended parameters
4. The test succeeded, got back the console

Did four runs and I got back the console each time

However with the provided binaries

Step 3 (stress-ng) failed for me once in two runs

Thanks for the update. I'll clone the source as well and rebuild the binaries for testing.

I cloned kamal's repository and built a test kernel. The test kernel can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1573062/

balbirs and Mike, can you test this kernel and see if it still exhibits the bug?

------- Comment From <email address hidden> 2016-07-17 21:43 EDT-------
I tried the kernel at http://people.canonical.com/~kamal/lp1573062/lp1573062.1/ and it worked fine for me

------- Comment From <email address hidden> 2016-07-19 01:04 EDT-------
Looks like I got a failure with the run on http://people.canonical.com/~kamal/lp1573062/lp1573062.1/

But with my diff + 4.4.0 source from apt-source, I can always get the the following command to succeed.

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

I've tried three times with my diff (all success) and twice with the kernel @ ~kamal (one failure and one success). I've not tried the longer 7 hour run

------- Comment From <email address hidden> 2016-07-19 01:37 EDT-------
In the kern.log posted, it looks like the problem has moved to

rwsem_wake+0xcc/0x110
up_write+0x78/0x90
unlink_anon_vmas+0x15c/0x2c0

A bunch of threads are stuck on rwsem_wake -- spinning on the sem->wait_lock. I can see a whole bunch of exiting stress-ng-mmapf stuck on this lock, spinning. I'll double check this. Can we get a build with lockdep enabled? I am unable to reproduce this issue at my end with the diff applied on my machine at the moment

------- Comment From <email address hidden> 2016-07-19 19:51 EDT-------
I am cloning the sources to debug further

------- Comment From <email address hidden> 2016-07-19 23:52 EDT-------
I cloned the kernel from https://git.launchpad.net/~kamalmostafa/ubuntu/+source/linux/+git/xenial/log/?h=lp1573062 and built with the machine config specified from /boot/config. I also verified the diff matches my changes.

I ran

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

twice

Both the times, the test did the right thing. Could someone verify if

(a) The smaller subset works fine?
(b) The larger test fails, if so, can we get a run with lockdep

I was just testing for the command line above and I could see a difference with those patches.

------- Comment From <email address hidden> 2016-07-21 20:14 EDT-------
No, the diff matches, sorry for the confusion, but here is what I said

"I also verified the diff matches my changes"

In summary, here is what I did

1. cloned the sources
2. built locally on my machine
3. Ran stress-ng with recommended parameters
4. The test succeeded, got back the console

Did four runs and I got back the console each time

However with the provided binaries

Step 3 (stress-ng) failed for me once in two runs

------- Comment From <email address hidden> 2016-07-25 08:08 EDT-------
Strange, I am able to reproduce the issue with the provided binaries, but not when I build it. I am not doing a deb build, but just a make -j64 with the config from /boot for 4.4.0-28. The problem could be at my end, but I am a little concerned.

I also noticed that if I am interacting with the system during runs, it succeeds, frequently checking if the console is active (enters and control-o-h). I am going to see if I can get a repro again and debug further.

Default Comment by Bridge

------- Comment From <email address hidden> 2016-07-17 21:43 EDT-------
I tried the kernel at http://people.canonical.com/~kamal/lp1573062/lp1573062.1/ and it worked fine for me

------- Comment From <email address hidden> 2016-07-19 01:04 EDT-------
Looks like I got a failure with the run on http://people.canonical.com/~kamal/lp1573062/lp1573062.1/

But with my diff + 4.4.0 source from apt-source, I can always get the the following command to succeed.

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

I've tried three times with my diff (all success) and twice with the kernel @ ~kamal (one failure and one success). I've not tried the longer 7 hour run

------- Comment From <email address hidden> 2016-07-19 01:37 EDT-------
In the kern.log posted, it looks like the problem has moved to

rwsem_wake+0xcc/0x110
up_write+0x78/0x90
unlink_anon_vmas+0x15c/0x2c0

A bunch of threads are stuck on rwsem_wake -- spinning on the sem->wait_lock. I can see a whole bunch of exiting stress-ng-mmapf stuck on this lock, spinning. I'll double check this. Can we get a build with lockdep enabled? I am unable to reproduce this issue at my end with the diff applied on my machine at the moment

------- Comment From <email address hidden> 2016-07-19 19:51 EDT-------
I am cloning the sources to debug further

------- Comment From <email address hidden> 2016-07-19 23:52 EDT-------
I cloned the kernel from https://git.launchpad.net/~kamalmostafa/ubuntu/+source/linux/+git/xenial/log/?h=lp1573062 and built with the machine config specified from /boot/config. I also verified the diff matches my changes.

I ran

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

twice

Both the times, the test did the right thing. Could someone verify if

(a) The smaller subset works fine?
(b) The larger test fails, if so, can we get a run with lockdep

I was just testing for the command line above and I could see a difference with those patches.

------- Comment From <email address hidden> 2016-07-21 20:14 EDT-------
No, the diff matches, sorry for the confusion, but here is what I said

"I also verified the diff matches my changes"

In summary, here is what I did

1. cloned the sources
2. built locally on my machine
3. Ran stress-ng with recommended parameters
4. The test succeeded, got back the console

Did four runs and I got back the console each time

However with the provided binaries

Step 3 (stress-ng) failed for me once in two runs

------- Comment From <email address hidden> 2016-07-25 08:08 EDT-------
Strange, I am able to reproduce the issue with the provided binaries, but not when I build it. I am not doing a deb build, but just a make -j64 with the config from /boot for 4.4.0-28. The problem could be at my end, but I am a little concerned.

I also noticed that if I am interacting with the system during runs, it succeeds, frequently checking if the console is active (enters and control-o-h). I am going to see if I can get a repro again and debug further.

------- Comment From <email address hidden> 2016-07-25 09:09 EDT-------
In the meanwhile, any updates on the bisect? I was hoping we could do both things (RCA and bisect) in parallel
...

The bisect was stopped because the first good kernel was found to actually be bad. We can restart the bisect if we can find a good and bad kernel again.

@Mike, could you test the v4.7 kernel and see if the bug is fixed in it? It can be downloaded from:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.7/

------- Comment From <email address hidden> 2016-07-17 21:43 EDT-------
I tried the kernel at http://people.canonical.com/~kamal/lp1573062/lp1573062.1/ and it worked fine for me

------- Comment From <email address hidden> 2016-07-19 01:04 EDT-------
Looks like I got a failure with the run on http://people.canonical.com/~kamal/lp1573062/lp1573062.1/

But with my diff + 4.4.0 source from apt-source, I can always get the the following command to succeed.

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

I've tried three times with my diff (all success) and twice with the kernel @ ~kamal (one failure and one success). I've not tried the longer 7 hour run

------- Comment From <email address hidden> 2016-07-19 01:37 EDT-------
In the kern.log posted, it looks like the problem has moved to

rwsem_wake+0xcc/0x110
up_write+0x78/0x90
unlink_anon_vmas+0x15c/0x2c0

A bunch of threads are stuck on rwsem_wake -- spinning on the sem->wait_lock. I can see a whole bunch of exiting stress-ng-mmapf stuck on this lock, spinning. I'll double check this. Can we get a build with lockdep enabled? I am unable to reproduce this issue at my end with the diff applied on my machine at the moment

------- Comment From <email address hidden> 2016-07-19 19:51 EDT-------
I am cloning the sources to debug further

------- Comment From <email address hidden> 2016-07-19 23:52 EDT-------
I cloned the kernel from https://git.launchpad.net/~kamalmostafa/ubuntu/+source/linux/+git/xenial/log/?h=lp1573062 and built with the machine config specified from /boot/config. I also verified the diff matches my changes.

I ran

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

twice

Both the times, the test did the right thing. Could someone verify if

(a) The smaller subset works fine?
(b) The larger test fails, if so, can we get a run with lockdep

I was just testing for the command line above and I could see a difference with those patches.

------- Comment From <email address hidden> 2016-07-21 20:14 EDT-------
No, the diff matches, sorry for the confusion, but here is what I said

"I also verified the diff matches my changes"

In summary, here is what I did

1. cloned the sources
2. built locally on my machine
3. Ran stress-ng with recommended parameters
4. The test succeeded, got back the console

Did four runs and I got back the console each time

However with the provided binaries

Step 3 (stress-ng) failed for me once in two runs

------- Comment From <email address hidden> 2016-07-25 08:08 EDT-------
Strange, I am able to reproduce the issue with the provided binaries, but not when I build it. I am not doing a deb build, but just a make -j64 with the config from /boot for 4.4.0-28. The problem could be at my end, but I am a little concerned.

I also noticed that if I am interacting with the system during runs, it succeeds, frequently checking if the console is active (enters and control-o-h). I am going to see if I can get a repro again and debug further.

------- Comment From <email address hidden> 2016-07-25 09:09 EDT-------
In the meanwhile, any updates on the bisect? I was hoping we could do both things (RCA and bisect) in parallel
...

mainline kernel 4.7 failed.

Output of 4.7 mainline kernel testing failure

------- Comment From <email address hidden> 2016-07-17 21:43 EDT-------
I tried the kernel at http://people.canonical.com/~kamal/lp1573062/lp1573062.1/ and it worked fine for me

------- Comment From <email address hidden> 2016-07-19 01:04 EDT-------
Looks like I got a failure with the run on http://people.canonical.com/~kamal/lp1573062/lp1573062.1/

But with my diff + 4.4.0 source from apt-source, I can always get the the following command to succeed.

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

I've tried three times with my diff (all success) and twice with the kernel @ ~kamal (one failure and one success). I've not tried the longer 7 hour run

------- Comment From <email address hidden> 2016-07-19 01:37 EDT-------
In the kern.log posted, it looks like the problem has moved to

rwsem_wake+0xcc/0x110
up_write+0x78/0x90
unlink_anon_vmas+0x15c/0x2c0

A bunch of threads are stuck on rwsem_wake -- spinning on the sem->wait_lock. I can see a whole bunch of exiting stress-ng-mmapf stuck on this lock, spinning. I'll double check this. Can we get a build with lockdep enabled? I am unable to reproduce this issue at my end with the diff applied on my machine at the moment

------- Comment From <email address hidden> 2016-07-19 19:51 EDT-------
I am cloning the sources to debug further

------- Comment From <email address hidden> 2016-07-19 23:52 EDT-------
I cloned the kernel from https://git.launchpad.net/~kamalmostafa/ubuntu/+source/linux/+git/xenial/log/?h=lp1573062 and built with the machine config specified from /boot/config. I also verified the diff matches my changes.

I ran

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

twice

Both the times, the test did the right thing. Could someone verify if

(a) The smaller subset works fine?
(b) The larger test fails, if so, can we get a run with lockdep

I was just testing for the command line above and I could see a difference with those patches.

------- Comment From <email address hidden> 2016-07-21 20:14 EDT-------
No, the diff matches, sorry for the confusion, but here is what I said

"I also verified the diff matches my changes"

In summary, here is what I did

1. cloned the sources
2. built locally on my machine
3. Ran stress-ng with recommended parameters
4. The test succeeded, got back the console

Did four runs and I got back the console each time

However with the provided binaries

Step 3 (stress-ng) failed for me once in two runs

------- Comment From <email address hidden> 2016-07-25 08:08 EDT-------
Strange, I am able to reproduce the issue with the provided binaries, but not when I build it. I am not doing a deb build, but just a make -j64 with the config from /boot for 4.4.0-28. The problem could be at my end, but I am a little concerned.

I also noticed that if I am interacting with the system during runs, it succeeds, frequently checking if the console is active (enters and control-o-h). I am going to see if I can get a repro again and debug further.

------- Comment From <email address hidden> 2016-07-25 09:09 EDT-------
In the meanwhile, any updates on the bisect? I was hoping we could do both things (RCA and bisect) in parallel
...

I built a Xenial test kernel with the latest patch posted to mm by balbirs. The test kernel can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1573062/

@Mike, can you test this kernel when you have a chance?

Balbir,

Both Mike and Joe are out the rest of this week on holiday. Any chance you could give this kernel a spin?

                 Michael

I just looked at the directory and I can find just the arm64 kernel. Could you please confirm if I am looking at the right thing and at the right place?

Sorry I built arm64. I'll build ppc64el now.

The test kernel is now available in the same location:
http://kernel.ubuntu.com/~jsalisbury/lp1573062/

Also note, with this test kernel you have to install both the linux-image and linux-image-extra .deb packages.

At my end, I ran two runs with success. More runs in progress

Had several other runs of success. I would like to see runs from others as well.

Can I quickly check if the oom_reaper patches are there in the built kernel or is it just the fix I posted?

The test kernel in comment #103 did not have the oom_reaper patches. It only had the fix you posted.

I have had 5 successful runs so far with below kernel. however I keep see OOM call traces.

root@ltc-haba1:~/log# uname -a
Linux ltc-haba1 4.4.0-34-generic #53~lp1573062PATCHED SMP Wed Aug 10 23:59:53 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux

I had this fail on 2 openpower boxes. I have attached the output from the test.

I have been trying to recreate this issue on my Haba-LC system and not able to do so.
The difference I see in total memory installed on my system against your system i.e 32Gig.

Total memory is 256 GiB -----------------------------> my Haba-LC.
Constant run time is 300 seconds per stressor
Variable run time is 2860 seconds per stressor
Estimated total run time is 396 minutes

Is there any way I can run the test for less memory config, the system I have got is 256Gig mem.

Kalpana:

The above failed test was run on only 1 of the possible 4 machines we have been testing with for months now. The other openpower server is a Habanero with 256G. I am running tests on that as we speak and going to include the kern.log as well.

Just to clarify, the memory test I'm running is /usr/lib/plainbox-provider-checkbox/bin/memory_stress_ng

I am unable to reproduce the failure either, but your system with 32G and 128 threads seems like the test would start 128 hogs each hogging up 32GB. How much swap do you have on them? Could you post the dmesg to see what failed and the logs around it? It looks like the stack stressor failed.

This is a failed test from today on the Habanero. Find the test output and kern.log attached:

ubuntu@binacle:~/4.4.0-34-generic-53~lp1573062PATCHED$ free -m
              total used free shared buff/cache available
Mem: 261533 621 259912 20 1000 259938
Swap: 8191 0 8191

kern.log from the Habanero 4.4.0-34-generic-53~lp1573062PATCHED test

This is a failed test from today on the Firestone. Find the test output and kern.log attached:

ubuntu@gulpin:~/4.4.0-34-generic-53~lp1573062PATCHED$ free -m
              total used free shared buff/cache available
Mem: 32565 342 30092 21 2131 30918
Swap: 8191 0 8191

kern.log from the Firestone 4.4.0-34-generic-53~lp1573062PATCHED test

These logs are something I've not seen here in my testing. This shows that we are stuck doing an up_write() on root->rwsem in the anon_vma path. It looks like we are contending on the rwsem's sem->wait_lock. I don't have a reproduction of this issue, it will be interesting to examine what is causing the heavy contention

Out of the 4 tests last night (Habanero(NV), Firstone(NV), Tuleta(NV), Alpine(VM) only the Firestone failed. Attached is the output from the test, kern.log and dmesg output I was running in a while loop during testing.

kern.log from failed test on Firestone

dmesg from failed test on Firestone

Just an additional data point, I ran this test (using Mike's script in a loop) on other architectures:

amd64 bare metal: 30 times 4.4.0, no failures
amd64 virtual machine: 30 times, 4.4.0, no failures

s390 4GB RAM: 40 times on both 4.4.0-31 and 4.7.0, no failures
s390 20GB RAM: 40 times on both 4.4.0-31 and 4.7.0, no failures.

I'm still running on arm64 and waiting on final results.

Thanks Jeff. I see that the ARM64 might have failed - https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1610320

Can we know the git commit id that fixed the ARM64 failure in mainline?

BTW, could you please share the full machine configurations - threads+RAM+swap for each of the other architectures you ran this on?

This comment is to inform you that mirroring has been disabled between Launchpad bug 1573062 and LTC bug 141723 due to the size of this LP bug attachments.

If you want to make this LP bug bridgeable again, I suggest you to remove unwanted attachments or perhaps to compress some of the existing attachments (several large uncompressed text log files currently exist).

I just posted another patch @ http://<email address hidden>/msg1219903.html, I am testing this patch at the moment.

Joseph: Is it possible for you to post test kernel with Balbir's patches ?
and place them at earlier location - http://kernel.ubuntu.com/~jsalisbury/lp1573062/

Attached is the cultivation of all the kernel testing we have done.

Thank you for the excellent summary. Questions

1. Can we get the configurations of the machines.
2. The first column is the number of times the test ran?
3. I see that 4.4.0-31-generic-50-Ubuntu passed on all machines across several runs, is that true?
4. Did any of the tests result in system hang? Can we find out from the summary?

Would it be fair to assume that tests when run against 4.4.0-31-generic-50-Ubuntu would pass again and we should work off of that?

My interest is in 4.4.0-34-generic-53~lp1573062PATCHED, With that I notice that gulpin saw failures in mmapfork and probably a hang there -- for which I posted a scheduler try_to_wake_up fix upstream. Generally binacle faces brk stress test failures -- it will be interesting to see its machine configuration and why the test failed

One observation at my end is that we should reboot between runs as I think some tests can kill important tasks in the system and I am not sure if there is a guarantee that the system is able to carry on correct operation recovering from all the tasks being re-spawned after OOM for example.

Sorry, I should have taken out the 4.4.0-31 tests. That test was not using stress-ng but our original memory stress test.

Can we please get answers to 1, 2 and 4 for comment #128. Also Kalpana has a request for a new kernel build.

Thanks,

1. Can we get the configurations of the machines.

XML files for all 4 machines with hardware information from the certification process will be attached in the next few comments

2. The first column is the number of times the test ran?

Yes

4. Did any of the tests result in system hang? Can we find out from the summary?

All failed tests failed with a complete system hang. The only way to recover would be to reboot.

Balbir, Kalpana,

I can build the kernel for you. Can you post a list of all the patches you would like included? That way we ensure they are all included.

Can we build the latest 4.8, may be we should wait for 4.8-rc7. I've got all the fixes upstream, with the latest being 135e8c9250dd5c8c9aae5984fde6f230d0cbfeaf

Sure, we can wait for 4.8-rc7. It should be available on Monday. I'll post a link to the download location once it's available.

FYI, I think the patch made it to 4.4 stable as well

Balibir, the v4.8 final kernel is now available and can be downloaded from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.8/

The latest upstream stable 4.4 kernel is also avaiable from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4.23

output of stress_ng test running on Power 8 Tuleta LPAR

dmesg output while stress_ng test was running on Power 8 Tuleta LPAR

`ps -ax` output while stress_ng test was running on Power 8 Tuleta LPAR

We're getting zombies here which aren't being reaped:

130428 ? Z 0:00 [stress-ng-brk] <defunct>
130432 ? Z 0:00 [stress-ng-brk] <defunct>
130434 ? Z 0:00 [stress-ng-brk] <defunct>
130436 ? Z 0:00 [stress-ng-brk] <defunct>

The reason for this is that memory stressors like brk have a parent that forks off a child. The child performs the stressing and if it gets OOM'd the parent can spawn off another stressor. So I think the SIGKILL on the stress-ng brk stressor is killing the parent bug the child (which is still holding onto a load of memory on the heap) is not being waited for and hence is in a memory hogging zombie state. We may be in a pathologically memory hogging state because the zombies may be holding brk regions that are swapped out to disk due to memory pressure and we're hitting a low-memory state which is not being cleared up.

I suggest modifying the test bash script as follows:

1. run stress-ng with -k flag (so that all the processes have the same stress-ng name)
2. kill with ALRM first
3. then kill with KILL all the stress-ng processes after a small grace period.
4. report on unkillable stressors

refer to the changes I made to https://launchpadlibrarian.net/296974522/disk_stress_ng

------- Comment From <email address hidden> 2016-12-20 22:55 EDT-------
Could the Ubuntu team check if this is still an issue with the 4.8 kernel?

Ok, so we're now using stress-ng 0.07.21 which resolved a lot of issues and have also adopted Colin's suggestions into our wrapper script to modify how stress-ng is being called and culled.

This seems to resolve the problems that led to this.

This issue, or similar symptoms doesn't seem to be fixed. This is with stock kernel, stock stress-ng but with the suggested changes to the memory_stress_ng test script. The machine locks up completely maybe 1 out of 5-10 runs. See attached dmesg for errors.

Kernel: 4.4.0-64-generic-85-Ubuntu
stress-ng Version: 0.07.21-1~ppa

OK, I have a Firestone available, so I'm re-trying this with a cycle of 10 runs to see what happens. Currently using 16.04 w/ HWE-Edge (4.10)

OK... so I tried three runs of the stress-ng memory test (that has been used successfully on every other system we certify for 16.04) and so far, two of three runs have resulted in the system locking up and a bunch of stack trace data dumped to console.

In both cases, I had to power cycle the system to reboot it into a usable state.

All three attempts were 16.04 w/ hwe-edge (4.10) deployed by MAAS.

System info:

Linux oil-entei 4.10.0-20-generic #22~16.04.1-Ubuntu SMP Thu Apr 20 10:30:58 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux

stress-ng:
  Installed: 0.07.21-1~ppa
(We keep an updated version of stress-ng in the cert PPA, it's not modified from Colin's code)

I've attached logs (kernel and syslog, other info) in a tarball to this.

resetting all the tasks to confirmed, rather than Incomplete as I've tested and am still waiting for a resolution or whatever.

This bug was nominated against a series that is no longer supported, ie yakkety. The bug task representing the yakkety nomination is being closed as Won't Fix.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.