More migrations with constant load

Default Comment by Bridge

Default Comment by Bridge

Default Comment by Bridge

I built a Zesty test kernel with commit 05b40e0577. The test kernel can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1713576

Can you test this kernel and see if it resolves this bug?

------- Comment From <email address hidden> 2017-09-07 17:48 EDT-------
(In reply to comment #8)
> I built a Zesty test kernel with commit 05b40e0577. The test kernel can be
> downloaded from:
>
> http://kernel.ubuntu.com/~jsalisbury/lp1713576
>
> Can you test this kernel and see if it resolves this bug?

Thanks, but I am having trouble with the perf packages.. any idea what is wrong?

# uname -a
Linux ltc-boston114 4.10.0-33-generic #37~lp1713576 SMP Wed Sep 6 15:50:28 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux

# dpkg -i /home/migration_test_build/linux-tools-common_4.10.0-33.37~lp1713576_all.deb
(Reading database ... 99780 files and directories currently installed.)
Preparing to unpack .../linux-tools-common_4.10.0-33.37~lp1713576_all.deb ...
Unpacking linux-tools-common (4.10.0-33.37~lp1713576) over (4.10.0-33.37~lp1713576) ...
Setting up linux-tools-common (4.10.0-33.37~lp1713576) ...
# dpkg -i /home/migration_test_build/linux-cloud-tools-common_4.10.0-33.37~lp1713576_all.deb
(Reading database ... 99780 files and directories currently installed.)
Preparing to unpack .../linux-cloud-tools-common_4.10.0-33.37~lp1713576_all.deb ...
Unpacking linux-cloud-tools-common (4.10.0-33.37~lp1713576) over (4.10.0-33.37~lp1713576) ...
Setting up linux-cloud-tools-common (4.10.0-33.37~lp1713576) ...

# perf list
The program 'perf' is currently not installed. You can install it by typing:
apt install linux-tools-common

# apt install linux-tools-common
Reading package lists... Done
Building dependency tree
Reading state information... Done
linux-tools-common is already the newest version (4.10.0-33.37~lp1713576).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

---

Ebizzy does show a 42% improvement on average so that is probably enough, although I was hoping to validate the sched migration counts are lower.

Did you happen to install both the linux-image and linux-image-extra .deb packages?

------- Comment From <email address hidden> 2017-09-08 15:59 EDT-------
Yes I installed all 9 .debs, I just re-dowloaded, installed, booted but still no luck.

You may have to install the linux-tools-common with the stock kernel before installing the test kernel?

------- Comment From <email address hidden> 2017-09-11 15:49 EDT-------
Thanks, I was able to make perf work by uninstalling the "linux-tools-common" test .deb and installing the stock version.

This test build resolves the problem by significantly reducing sched migrations in the under utilized case which improves performance by 30-40% for this ebizzy testcase:

stock kernel (4.10.0-33-generic) :
6,010 sched:sched_migrate_task

test kernel (4.10.0-33-generic #37~lp1713576) :
676 sched:sched_migrate_task

Testcase was:
perf stat -a -r 5 -e sched:sched_migrate_task ./ebizzy -t 35 -S 100

where ebizzy was using 35 threads on a system w/ 128 logical cpus total

Thanks for the update. Can this issue now be closed?

------- Comment From <email address hidden> 2017-09-12 10:08 EDT-------
(In reply to comment #14)
> Thanks for the update. Can this issue now be closed?

Should we wait until it is in an official build?

@jhopper, yes, I will submit a SRU request for commit 05b40e0577 to be included in 17.04.

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-zesty' to 'verification-done-zesty'. If the problem still exists, change the tag 'verification-needed-zesty' to 'verification-failed-zesty'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Hello IBM,

Could you please verify the fix with the zesty kernel currently in -proposed?

Thank you!

This bug was fixed in the package linux - 4.10.0-38.42

---------------
linux (4.10.0-38.42) zesty; urgency=low

  * linux: 4.10.0-38.42 -proposed tracker (LP: #1722330)

  * Controller lockup detected on ProLiant DL380 Gen9 with P440 Controller
    (LP: #1720359)
    - scsi: hpsa: limit transfer length to 1MB

  * [Dell Docking IE][0bda:8153] Realtek USB Ethernet leads to system hang
    (LP: #1720977)
    - r8152: fix the list rx_done may be used without initialization

  * Touchpad not detected in Lenovo X1 Yoga / Yoga 720-15IKB (LP: #1700657)
    - mfd: intel-lpss: Add missing PCI ID for Intel Sunrise Point LPSS devices

  * Add installer support for Broadcom BCM573xx network drivers. (LP: #1720466)
    - d-i: Add bnxt_en to nic-modules.

  * CVE-2017-1000252
    - KVM: VMX: Do not BUG() on out-of-bounds guest IRQ

  * CVE-2017-10663
    - f2fs: sanity check checkpoint segno and blkoff

  * xfstest sanity checks on seek operations fails (LP: #1696049)
    - xfs: fix off-by-one on max nr_pages in xfs_find_get_desired_pgoff()

  * [P9, Power NV][ WSP][Ubuntu 16.04.03] : perf hw breakpoint command results
    in call traces and system goes for reboot. (LP: #1706033)
    - powerpc/64s: Handle data breakpoints in Radix mode

  * 5U84 - ses driver isn't binding right - cannot blink lights on 1 of the 2
    5u84 (LP: #1693369)
    - scsi: ses: do not add a device to an enclosure if enclosure_add_links()
      fails.

  * Vlun resize request could fail with cxlflash driver (LP: #1713575)
    - scsi: cxlflash: Fix vlun resize failure in the shrink path

  * More migrations with constant load (LP: #1713576)
    - sched/fair: Prefer sibiling only if local group is under-utilized

  * New PMU fixes for marked events. (LP: #1716491)
    - powerpc/perf: POWER9 PMU stops after idle workaround

  * CVE-2017-14340
    - xfs: XFS_IS_REALTIME_INODE() should be false if no rt device present

  * [Zesty][Yakkety] rtl8192e bug fixes (LP: #1698470)
    - staging: rtl8192e: rtl92e_fill_tx_desc fix write to mapped out memory.
    - staging: rtl8192e: fix 2 byte alignment of register BSSIDR.
    - staging: rtl8192e: rtl92e_get_eeprom_size Fix read size of EPROM_CMD.
    - staging: rtl8192e: GetTs Fix invalid TID 7 warning.

  * Stranded with ENODEV after mdadm --readonly (LP: #1706243)
    - md: MD_CLOSING needs to be cleared after called md_set_readonly or
      do_md_stop

  * multipath -ll is not showing the disks which are actually multipath
    (LP: #1718397)
    - fs: aio: fix the increment of aio-nr and counting against aio-max-nr

  * ETPS/2 Elantech Touchpad inconsistently detected (Gigabyte P57W laptop)
    (LP: #1594214)
    - Input: i8042 - add Gigabyte P57 to the keyboard reset table

  * CVE-2017-10911
    - xen-blkback: don't leak stack data via response ring

  * CVE-2017-11176
    - mqueue: fix a use-after-free in sys_mq_notify()

  * implement 'complain mode' in seccomp for developer mode with snaps
    (LP: #1567597)
    - Revert "UBUNTU: SAUCE: seccomp: log actions even when audit is disabled"
    - seccomp: Provide matching filter for introspection
    - seccomp: Sysctl to display available actions
    - seccomp: Operation for checking if an a...

------- Comment From <email address hidden> 2017-10-30 16:55 EDT-------
Thanks, I verified the fix in -proposed dramatically reduces unnecessary migrations as expected and improves performance using an underutilized ebizzy testcase.

with previous kernel:
4.10.0-34-generic
5,640 sched:sched_migrate_task

with -proposed kernel:
4.10.0-38-generic
516 sched:sched_migrate_task

testcase was run on a 32core system:
# perf stat -a -r 5 -e sched:sched_migrate_task ./ebizzy -t 35 -S 100