SRU: walker list corruption while being intensively stressed

Fix will land in Linux 4.4 Ubuntu Xenial automatically, commit c6ff5268293ef98e48a99597e765ffc417e39fa5

Wily SRU submitted to kernel team mailing list, https://lists.ubuntu.com/archives/kernel-team/2015-December/067287.html

Looks like the fix triggered another issue, so we need to see how upstream fixes this one.

Kernel test robot picked up a bug from this fix:

FYI, we noticed the below changes on

https://github.com/0day-ci/linux
Herbert-Xu/rhashtable-Fix-walker-list-corruption/20151216-164833
commit f9f51b8070be3e829100614a7372b219723b864f ("rhashtable: Fix walker
list corruption")

[ 8.933376] ===============================
[ 8.933376] ===============================
[ 8.934629] [ INFO: suspicious RCU usage. ]
[ 8.934629] [ INFO: suspicious RCU usage. ]
[ 8.935941] 4.4.0-rc3-00995-gf9f51b8 #2 Not tainted
[ 8.935941] 4.4.0-rc3-00995-gf9f51b8 #2 Not tainted
[ 8.937494] -------------------------------
[ 8.937494] -------------------------------
[ 8.938818] lib/rhashtable.c:504 suspicious
rcu_dereference_protected() usage!
[ 8.938818] lib/rhashtable.c:504 suspicious
rcu_dereference_protected() usage!
[ 8.941705]
[ 8.941705] other info that might help us debug this:
[ 8.941705]
[ 8.941705]
[ 8.941705] other info that might help us debug this:
[ 8.941705]
[ 8.944161]
[ 8.944161] rcu_scheduler_active = 1, debug_locks = 0
[ 8.944161]
[ 8.944161] rcu_scheduler_active = 1, debug_locks = 0
[ 8.946244] 1 lock held by swapper/0/1:
[ 8.946244] 1 lock held by swapper/0/1:
[ 8.947463] #0:
[ 8.947463] #0: (
(&(&ht->lock)->rlock&(&ht->lock)->rlock){+.+...}){+.+...}, at: , at:
[<ffffffff814b8900>] rhashtable_walk_init+0x70/0x150
[<ffffffff814b8900>] rhashtable_walk_init+0x70/0x150
[ 8.950428]
[ 8.950428] stack backtrace:
[ 8.950428]
[ 8.950428] stack backtrace:
[ 8.951770] CPU: 0 PID: 1 Comm: swapper/0 Not tainted
4.4.0-rc3-00995-gf9f51b8 #2
[ 8.951770] CPU: 0 PID: 1 Comm: swapper/0 Not tainted
4.4.0-rc3-00995-gf9f51b8 #2
[ 8.954245] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS Debian-1.8.2-1 04/01/2014
[ 8.954245] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS Debian-1.8.2-1 04/01/2014
[ 8.956973] 0000000000000001
[ 8.956973] 0000000000000001 ffff880078393d30 ffff880078393d30
ffffffff81493238 ffffffff81493238 ffff88007838c040 ffff88007838c040

[ 8.959333] ffff880078393d60
[ 8.959333] ffff880078393d60 ffffffff8112cb9f ffffffff8112cb9f
ffff880078393da0 ffff880078393da0 ffffffff83e9d6c0 ffffffff83e9d6c0

[ 8.961684] ffffffff83e9d7f0
[ 8.961684] ffffffff83e9d7f0 ffff880061720e00 ffff880061720e00
ffff880078393d90 ffff880078393d90 ffffffff814b89c8 ffffffff814b89c8

[ 8.964148] Call Trace:
[ 8.964148] Call Trace:
[ 8.964955] [<ffffffff81493238>] dump_stack+0x7c/0xb4
[ 8.964955] [<ffffffff81493238>] dump_stack+0x7c/0xb4
[ 8.966728] [<ffffffff8112cb9f>] lockdep_rcu_suspicious+0x14f/0x1c0
[ 8.966728] [<ffffffff8112cb9f>] lockdep_rcu_suspicious+0x14f/0x1c0
[ 8.968753] [<ffffffff814b89c8>] rhashtable_walk_init+0x138/0x150
[ 8.968753] [<ffffffff814b89c8>] rhashtable_walk_init+0x138/0x150
[ 8.970567] [<ffffffff815021d8>] test_bucket_stats+0x22/0x17d
[ 8.970567] [<ffffffff815021d8>] test_bucket_stats+0x22/0x17d
[ 8.972682] [<ffffffff82dda0fa>] test_rhashtable+0xe0/0x12ac
[ 8.972682] [<ffffffff...

Seems that upstream commit 179ccc0a73641ffd24e44ff10a7bd494efe98d8d ("rhashtable: Kill harmless RCU warning in rhashtable_walk_init") is also required to stop the RCU warning in rhashtable_walk_init

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-wily' to 'verification-done-wily'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

A better reproducer is running stress-ng --procfs 0 on a multi-core machine. Without the fix, it oopses in less than a second. With the fix, it works perfectly, no oopsing.

Tested on 4.2.0-24-generic #29-Ubuntu, ran soak test for 600 seconds on an 8 proc Xeon box:

stress-ng: info: [3044] successful run completed in 600.23s (10 mins, 0.23 secs)
stress-ng: info: [3044] stressor bogo ops real time usr time sys time bogo ops/s bogo ops/s
stress-ng: info: [3044] (secs) (secs) (secs) (real time) (usr+sys time)
stress-ng: info: [3044] procfs 8 600.00 151.83 4646.48 0.01 0.00
stress-ng: info: [3044] procfs:

This bug was fixed in the package linux - 4.3.0-6.17

---------------
linux (4.3.0-6.17) xenial; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1532958

  [ Eric Dumazet ]

  * SAUCE: (noup) net: fix IP early demux races
    - LP: #1526946

  [ Guilherme G. Piccoli ]

  * SAUCE: powerpc/eeh: Validate arch in eeh_add_device_early()
    - LP: #1486180

  [ Hui Wang ]

  * [Config] CONFIG_I2C_DESIGNWARE_BAYTRAIL=y, CONFIG_IOSF_MBI=y
    - LP: #1527096

  [ Jann Horn ]

  * ptrace: being capable wrt a process requires mapped uids/gids
    - LP: #1527374

  [ Serge Hallyn ]

  * SAUCE: add a sysctl to disable unprivileged user namespace unsharing

  [ Tim Gardner ]

  * [Config] CONFIG_ZONE_DEVICE=y for amd64
  * [Config] CONFIG_VIRTIO_BLK=y, CONFIG_VIRTIO_NET=y for s390
    - LP: #1532886

  [ Upstream Kernel Changes ]

  * rhashtable: Fix walker list corruption
    - LP: #1526811
  * rhashtable: Kill harmless RCU warning in rhashtable_walk_init
    - LP: #1526811
  * ovl: fix permission checking for setattr
    - LP: #1528904
    - CVE-2015-8660

 -- Tim Gardner <email address hidden> Thu, 17 Dec 2015 05:34:47 -0700

Fix released in 4.2.0-27.32