Ubuntu18.04.1: cpuidle: powernv: Fix promotion from snooze if next state disabled (performance)

Bug #1790602 reported by bugproxy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
The Ubuntu-power-systems project
Fix Released
Critical
Canonical Kernel Team
linux (Ubuntu)
Fix Released
Critical
Canonical Kernel Team
Bionic
Fix Released
Critical
Unassigned

Bug Description

== Comment: #0 - Michael Ranweiler <email address hidden> - 2018-09-04 00:50:14 ==

The CPUs remain in snooze state on an idle system when only the shallow states disabled resulting in increased power consumption.

The commit 78eaa10f027c ("cpuidle: powernv/pseries: Auto-promotion of snooze to deeper idle state") introduced a timeout for the snooze idle state so that it could be eventually be promoted to a deeper idle state. The snooze timeout value is static and set to the target residency of the next idle state, which would train the cpuidle governor to pick the next idle state eventually.

The unfortunate side-effect of this is that if the next idle state(s) is disabled, the CPU will forever remain in snooze, despite the fact that the system is completely idle, and other deeper idle states are available.

This patch fixes the issue by dynamically setting the snooze timeout to the target residency of the next enabled state on the device.

Before Patch:
      POWER8 : Only nap disabled.
      $ cpupower monitor sleep 30
      sleep took 30.01297 seconds and exited with status 0
                    |Idle_Stats
      PKG |CORE|CPU | snoo | Nap | Fast
         0| 8| 0| 96.41| 0.00| 0.00
         0| 8| 1| 96.43| 0.00| 0.00
         0| 8| 2| 96.47| 0.00| 0.00
         0| 8| 3| 96.35| 0.00| 0.00
         0| 8| 4| 96.37| 0.00| 0.00
         0| 8| 5| 96.37| 0.00| 0.00
         0| 8| 6| 96.47| 0.00| 0.00
         0| 8| 7| 96.47| 0.00| 0.00

POWER9: Shallow states (stop0lite, stop1lite, stop2lite, stop0, stop1, stop2) disabled:
      $ cpupower monitor sleep 30
      sleep took 30.05033 seconds and exited with status 0
                    |Idle_Stats
      PKG |CORE|CPU | snoo | stop | stop | stop | stop | stop | stop | stop | stop
         0| 16| 0| 89.79| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
         0| 16| 1| 90.12| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
         0| 16| 2| 90.21| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
         0| 16| 3| 90.29| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00

After Patch:
      POWER8 : Only nap disabled.
      $ cpupower monitor sleep 30
      sleep took 30.01200 seconds and exited with status 0
                    |Idle_Stats
      PKG |CORE|CPU | snoo | Nap | Fast
         0| 8| 0| 16.58| 0.00| 77.21
         0| 8| 1| 18.42| 0.00| 75.38
         0| 8| 2| 4.70| 0.00| 94.09
         0| 8| 3| 17.06| 0.00| 81.73
         0| 8| 4| 3.06| 0.00| 95.73
         0| 8| 5| 7.00| 0.00| 96.80
         0| 8| 6| 1.00| 0.00| 98.79
         0| 8| 7| 5.62| 0.00| 94.17

      POWER9: Shallow states (stop0lite, stop1lite, stop2lite, stop0, stop1, stop2) disabled:

      $ cpupower monitor sleep 30
      sleep took 30.02110 seconds and exited with status 0
                    |Idle_Stats
      PKG |CORE|CPU | snoo | stop | stop | stop | stop | stop | stop | stop | stop
         0| 0| 0| 0.69| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 9.39| 89.70
         0| 0| 1| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.05| 93.21
         0| 0| 2| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 89.93
         0| 0| 3| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 93.26

This fix (commit 0a4ec6aa035a "cpuidle: powernv: Fix promotion from snooze if next state disabled") is in the -next branch of the powerpc tree and upstream:https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/cpuidle/cpuidle-powernv.c?id=0a4ec6aa035a52c422eceb2ed51ed88392a3d6c2

bugproxy (bugproxy)
tags: added: architecture-ppc64le bugnameltc-171057 severity-critical targetmilestone-inin---
Changed in ubuntu:
assignee: nobody → Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage)
affects: ubuntu → linux (Ubuntu)
bugproxy (bugproxy)
tags: added: targetmilestone-inin1804
removed: targetmilestone-inin---
Frank Heimes (fheimes)
Changed in ubuntu-power-systems:
importance: Undecided → Critical
assignee: nobody → Canonical Kernel Team (canonical-kernel-team)
Manoj Iyer (manjo)
Changed in linux (Ubuntu):
assignee: Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) → Canonical Kernel Team (canonical-kernel-team)
importance: Undecided → Critical
tags: added: triage-g
Revision history for this message
Manoj Iyer (manjo) wrote :

The patch is available in linux-stable, and could be fwd ported from 4.14, or backported from 4.17 to bionic.

-- linux-4.14.y --
3b185e667b52 cpuidle: powernv: Fix promotion from snooze if next state disabled

-- linux-4.17.y --
6de015f7f962 cpuidle: powernv: Fix promotion from snooze if next state disabled

Changed in linux (Ubuntu Bionic):
status: New → Triaged
importance: Undecided → Critical
Revision history for this message
Manoj Iyer (manjo) wrote :

http://kernel.ubuntu.com/~kamal/lp1790636-lp1790602-ppc/

Can you test this kernel and see if it resolves this bug?

Note about installing test kernels:
* If the test kernel is prior to 4.15(Bionic) you need to install the linux-image and linux-image-extra .deb packages.
* If the test kernel is 4.15(Bionic) or newer, you need to install the linux-modules, linux-modules-extra and linux-image-unsigned .deb packages.

Thanks in advance!

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2018-09-05 17:50 EDT-------
I recreated on the old kernel:
user@deb3qwsp1:~$ cat /proc/version
Linux version 4.15.0-33-generic (buildd@bos02-ppc64el-007) (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)) #36-Ubuntu SMP Wed Aug 15 13:42:39 UTC 2018
user@deb3qwsp1:~$ for i in `seq 1 6`; do sudo cpupower idle-set -d $i; done
Idlestate 1 disabled on CPU 0
Idlestate 1 disabled on CPU 1
Idlestate 1 disabled on CPU 2
<snip>
user@deb3qwsp1:~$ cpupower monitor sleep 30
sleep took 30.02607 seconds and exited with status 0
|Idle_Stats
PKG |CORE|CPU | snoo | stop | stop | stop | stop | stop | stop | stop | stop
0| 0| 0| 87.88| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 0| 1| 87.95| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 0| 2| 88.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 0| 3| 87.97| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
<snip>

And I booted the new kernel:
user@deb3qwsp1:~$ cat /proc/version
Linux version 4.15.0-35-generic (kamal@kathleen) (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)) #38 SMP Wed Sep 5 15:07:24 UTC 2018

But looks like cpupower is part of linux-tools-4.15.0-35-generic.... and just the linux-tools-common and linux-tools-hosts is available.

Can you please upload the linux-tools-4.15.0-35-generic_4.15.0-35.38_ppc64el.deb and I can finish the verification?

Changed in linux (Ubuntu Bionic):
status: Triaged → Fix Committed
tags: added: kernel-da-key
Revision history for this message
Kamal Mostafa (kamalmostafa) wrote :

I have added the package containing /usr/bin/cpupower (linux-tools-common_4.15.0-35.38_all.deb) to the test-kernel download folder, as well as the other arch-independent packages.

http://kernel.ubuntu.com/~kamal/lp1790636-lp1790602-ppc/linux-tools-common_4.15.0-35.38_all.deb

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-09-11 17:31 EDT-------
That package didn't have what I needed so I just symlink'd the -34 version to -35 - doesn't sound like there's much change.

With that and your kernel it looks good:

user@deb3qwsp1:~$ cat /proc/version
Linux version 4.15.0-35-generic (kamal@kathleen) (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)) #38 SMP Wed Sep 5 15:07:24 UTC 2018
user@deb3qwsp1:~$ cpupower monitor sleep 30
sleep took 30.02114 seconds and exited with status 0
|Idle_Stats
PKG |CORE|CPU | snoo | stop | stop | stop | stop | stop | stop | stop | stop
0| 0| 0| 0.09| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.12| 99.66
0| 0| 1| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 96.70
0| 0| 2| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 95.84
0| 0| 3| 2.54| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 96.71
0| 4| 4| 0.43| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.41| 94.66
0| 4| 5| 0.47| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 26.69| 68.68
0| 4| 6| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 95.84
0| 4| 7| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 95.84
<snip>
Thanks!

Revision history for this message
Brad Figg (brad-figg) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-bionic
Manoj Iyer (manjo)
Changed in ubuntu-power-systems:
status: New → Fix Committed
Changed in linux (Ubuntu):
status: New → Fix Committed
status: Fix Committed → New
status: New → Fix Committed
Revision history for this message
Mike Ranweiler (mranweil) wrote :

Looks good:
user@deb3qwsp1:~$ cat /proc/version
Linux version 4.15.0-35-generic (buildd@bos02-ppc64el-016) (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)) #38-Ubuntu SMP Wed Sep 12 10:29:44 UTC 2018
user@deb3qwsp1:~$ cpupower monitor sleep 30
sleep took 30.01774 seconds and exited with status 0
              |Idle_Stats
PKG |CORE|CPU | snoo | stop | stop | stop | stop | stop | stop | stop | stop
   0| 0| 0| 0.16| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.64| 99.11
   0| 0| 1| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 96.28
   0| 0| 2| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 96.28
   0| 0| 3| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 96.28
<snip>

tags: added: verification-done-bionic
removed: verification-needed-bionic
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (23.5 KiB)

This bug was fixed in the package linux - 4.15.0-36.39

---------------
linux (4.15.0-36.39) bionic; urgency=medium

  * CVE-2018-14633
    - iscsi target: Use hex2bin instead of a re-implementation

  * CVE-2018-17182
    - mm: get rid of vmacache_flush_all() entirely

linux (4.15.0-35.38) bionic; urgency=medium

  * linux: 4.15.0-35.38 -proposed tracker (LP: #1791719)

  * device hotplug of vfio devices can lead to deadlock in vfio_pci_release
    (LP: #1792099)
    - SAUCE: vfio -- release device lock before userspace requests

  * L1TF mitigation not effective in some CPU and RAM combinations
    (LP: #1788563)
    - x86/speculation/l1tf: Fix overflow in l1tf_pfn_limit() on 32bit
    - x86/speculation/l1tf: Fix off-by-one error when warning that system has too
      much RAM
    - x86/speculation/l1tf: Increase l1tf memory limit for Nehalem+

  * CVE-2018-15594
    - x86/paravirt: Fix spectre-v2 mitigations for paravirt guests

  * CVE-2017-5715 (Spectre v2 s390x)
    - KVM: s390: implement CPU model only facilities
    - s390: detect etoken facility
    - KVM: s390: add etoken support for guests
    - s390/lib: use expoline for all bcr instructions
    - s390: fix br_r1_trampoline for machines without exrl
    - SAUCE: s390: use expoline thunks for all branches generated by the BPF JIT

  * Ubuntu18.04.1: cpuidle: powernv: Fix promotion from snooze if next state
    disabled (performance) (LP: #1790602)
    - cpuidle: powernv: Fix promotion from snooze if next state disabled

  * Watchdog CPU:19 Hard LOCKUP when kernel crash was triggered (LP: #1790636)
    - powerpc: hard disable irqs in smp_send_stop loop
    - powerpc: Fix deadlock with multiple calls to smp_send_stop
    - powerpc: smp_send_stop do not offline stopped CPUs
    - powerpc/powernv: Fix opal_event_shutdown() called with interrupts disabled

  * Security fix: check if IOMMU page is contained in the pinned physical page
    (LP: #1785675)
    - vfio/spapr: Use IOMMU pageshift rather than pagesize
    - KVM: PPC: Check if IOMMU page is contained in the pinned physical page

  * Missing Intel GPU pci-id's (LP: #1789924)
    - drm/i915/kbl: Add KBL GT2 sku
    - drm/i915/whl: Introducing Whiskey Lake platform
    - drm/i915/aml: Introducing Amber Lake platform
    - drm/i915/cfl: Add a new CFL PCI ID.

  * CVE-2018-15572
    - x86/speculation: Protect against userspace-userspace spectreRSB

  * Support Power Management for Thunderbolt Controller (LP: #1789358)
    - thunderbolt: Handle NULL boot ACL entries properly
    - thunderbolt: Notify userspace when boot_acl is changed
    - thunderbolt: Use 64-bit DMA mask if supported by the platform
    - thunderbolt: Do not unnecessarily call ICM get route
    - thunderbolt: No need to take tb->lock in domain suspend/complete
    - thunderbolt: Use correct ICM commands in system suspend
    - thunderbolt: Add support for runtime PM

  * random oopses on s390 systems using NVMe devices (LP: #1790480)
    - s390/pci: fix out of bounds access during irq setup

  * [Bionic] Spectre v4 mitigation (Speculative Store Bypass Disable) support
    for arm64 using SMC firmware call to set a hardware chicken bit
    (LP: #1787993) // CVE-2018...

Changed in linux (Ubuntu Bionic):
status: Fix Committed → Fix Released
Changed in ubuntu-power-systems:
status: Fix Committed → Fix Released
Changed in linux (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Frank Heimes (fheimes) wrote :

Since we are at kernel level 4.18.0.8.9 in cosmic that incl. the fix:
$ git log --oneline | grep "cpuidle: powernv: Fix promotion from snooze if next state disabled"
0a4ec6a cpuidle: powernv: Fix promotion from snooze if next state disabled
$ git tag --contains 0a4ec6a
Ubuntu-4.18.0-7.8
Ubuntu-4.18.0-8.9
Ubuntu-4.18.0-9.10
v4.18
This can be marked as Fix Released for cosmic, too.

bugproxy (bugproxy)
tags: added: targetmilestone-inin18041
removed: targetmilestone-inin1804
Brad Figg (brad-figg)
tags: added: cscc
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.