[UBUNTU 20.04] KVM: s390: pv: don't allow userspace to set the clock under PV - qemu part

Bug #1999885 reported by bugproxy
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
Fix Released
High
Skipper Bug Screeners
qemu (Ubuntu)
Fix Released
Low
Ubuntu Server
Focal
Fix Released
Undecided
Unassigned
Jammy
Fix Released
Undecided
Unassigned
Kinetic
Fix Released
Undecided
Unassigned
Lunar
Fix Released
Low
Ubuntu Server

Bug Description

[ Impact ]

 * suspend and resume (and other situations that cause vm
   state change) lead to situations where the hipervisor
   wants to change the clock.
   Now the s390x clock can be considered staged with
   offsets potentially added at each level of virtualization.
   In the secure execution mode on s390x the hipervisor
   is not allowed to touch this, leading to wrong timing.

 * The fix skips changing the tod (time of day -> clock)
   on a kvm_s390_tod_vm_state_change

[ Test Plan ]

 * 1. Start SE guest using libvirt.
   2. Pause the guest using "virsh suspend", wait for a few
      seconds, resume using "virsh resume".
   3. Run "time sleep 1" in the guest.
   4. The sleep will sleep much longer than one second.

[ Where problems could occur ]

 * This is gladly very isolated to s390x clock handling, so
   that (and only that) would be the place to look out for
   problems or regressions.

[ Other info ]

 * s390x secure execution needs HW feature my machine lacks,
   this will need IBM to verify the case once accepted to proposed.

--- original report ---

Description: KVM: s390: pv: don't allow userspace to set the clock under PV

Symptom: Timer issues and RCU stalls after suspending and resuming an IBM
               Secure Execution guest

Problem: KVM and QEMU try to set the guest's TOD clock after resume
               under PV, even though that is not permitted under SE. Hence,
               their view of the guest clock may deviate from the ultravisor's,
               possibly causing KVM to re-dispatch the guest too late on clock
               comparator interrupts.

Solution: Don't set the clock after resume under PV. Note that kernel and
               QEMU patches are required in lockstep, to avoid a warning message
               in QEMU.

Reproduction: 1. Start SE guest using libvirt.
               2. Pause the guest using "virsh suspend", wait for a few
                  seconds, resume using "virsh resume".
               3. Run "time sleep 1" in the guest.
               4. The sleep will sleep much longer than one second.

Component: qemu

Upstream-ID: 38621181ae3cbec62e3490fbc14f6ac01642d07a

Preventive: yes

Author: Nico Boehr <email address hidden>

Please note that fixing the described problem requires patches for the kernel as well as for QEMU.
This bug covers the qemu part, whereas the required kernel part is described and handled in the following related bug:

   Bug 200889 - [UBUNTU 20.04] KVM: s390: pv: don't allow userspace to set the clock under PV - kernel part

Both parts, the kernel and the qemu patches should be applied / released at the same time to avoid problems resulting in the following warning message for customers:
   'warning: Unable to set KVM guest TOD clock: Operation not supported'

Related branches

bugproxy (bugproxy)
tags: added: architecture-s39064 bugnameltc-200901 severity-high targetmilestone-inin---
Changed in ubuntu:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
affects: ubuntu → linux (Ubuntu)
Frank Heimes (fheimes)
affects: linux (Ubuntu) → qemu (Ubuntu)
Changed in qemu (Ubuntu):
importance: Undecided → High
Changed in ubuntu-z-systems:
importance: Undecided → High
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
Changed in qemu (Ubuntu Lunar):
importance: High → Low
Changed in ubuntu-z-systems:
status: New → Triaged
Frank Heimes (fheimes)
Changed in qemu (Ubuntu Lunar):
assignee: Skipper Bug Screeners (skipper-screen-team) → Ubuntu Server (ubuntu-server)
Frank Heimes (fheimes)
Changed in qemu (Ubuntu Lunar):
status: New → In Progress
Changed in ubuntu-z-systems:
status: Triaged → In Progress
Revision history for this message
Frank Heimes (fheimes) wrote :

Meanwhile 7.2 landed in lunar-proposed:
qemu | 1:7.2+dfsg-3ubuntu1 | lunar-proposed
hence updating ticket status to Fix Committed.

Changed in qemu (Ubuntu Lunar):
status: In Progress → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (9.9 KiB)

This bug was fixed in the package qemu - 1:7.2+dfsg-4ubuntu1

---------------
qemu (1:7.2+dfsg-4ubuntu1) lunar; urgency=medium

  * Merge with Debian unstable (LP: #1993438), among many other fixes
    this resolvs these bugs:
    (LP: #1957924) - support for querying stats,
    (LP: #1853307) - Enhanced Interpretation for PCI Functions (s390x)
    (LP: #1959966) - guest dump encryption with customer keys (s390x)
    (LP: #1999885) - pv: don't allow userspace to set the clock under PV
    (LP: #1957924) - add filtering of statistics by target vCPU
    remaining changes:
    - qemu-kvm to systemd unit
      - d/qemu-kvm-init: script for QEMU KVM preparation modules, ksm,
        hugepages and architecture specifics
      - d/qemu-system-common.qemu-kvm.service: systemd unit to call
        qemu-kvm-init
      - d/qemu-system-common.install: install helper script
      - d/qemu-system-common.qemu-kvm.default: defaults for
        /etc/default/qemu-kvm
      - d/rules: call dh_installinit and dh_installsystemd for qemu-kvm
    - Distribution specific machine type
      (LP: 1304107 1621042 1776189 1761372 1761372 1776189)
      - d/p/ubuntu/define-ubuntu-machine-types.patch: define distro machine
        types containing release versioned machine attributes
      - d/qemu-system-x86.NEWS Info on fixed machine type defintions
        for host-phys-bits=true
      - Add an info about -hpb machine type in debian/qemu-system-x86.NEWS
      - ubuntu-q35 alias added to auto-select the most recent q35 ubuntu type
    - Enable nesting by default
      - d/p/ubuntu/enable-svm-by-default.patch: Enable nested svm by default
        in qemu64 on amd
        [ No more strictly needed, but required for backward compatibility ]
    - tolerate ipxe size change on migrations to >=18.04 (LP: 1713490)
      - d/p/ubuntu/pre-bionic-256k-ipxe-efi-roms.patch: old machine types
        reference 256k path
      - d/control-in: depend on ipxe-qemu-256k-compat-efi-roms to be able to
        handle incoming migrations from former releases.
    - d/qemu-system-x86.README.Debian: add info about updated nesting changes
    - Ease the use of module retention on upgrades (LP 1913421)
      - debian/qemu-block-extra.postinst: enable mount unit on install/upgrade
    - d/control-in: switch qemu-system-x86-xen to qemu-system-xen as this
      landed in Debian but under a different name.
    - Remaining GCC-12 FTBFS (LP 1988710 + LP 1921664)
      + d/p/u/qboot-Disable-LTO-for-ELF-binary-build-step.patch:
        fix qboot FTBFS with LTO
  * Dropped Changes [now part of upstream v7.2.0]
    - d/p/u/lp1994002-migration-Read-state-once.patch: Fix for libvirt
      error 'migration was active, but no RAM info was set' (LP 1994002)
    - d/p/u/ebpf-replace-deprecated-bpf_program__set_socket_filt.patch:
      Fix FTBFS with libbpf 1.0.1-2.
      + Header updates that were added as part of the libbpf fixes
        but not mentioned in changelog
    - d/p/u/lp-1981339-*: fix s390x system emulation (LP 1981339)
    - Fix I/O stalls when using NVMe storage (LP 1970737).
      + d/p/lp1970737-linux-aio-*.patch: Fix unbalanced plugged counter
        in laio_io_unplug.
    - SECURITY UPDATE...

Changed in qemu (Ubuntu Lunar):
status: Fix Committed → Fix Released
Revision history for this message
Frank Heimes (fheimes) wrote :

Hi Nico, do you know if backports exist to get 38621181ae3cbec62e3490fbc14f6ac01642d07a "KVM: s390: pv: don't allow userspace to set the clock under PV" applied to kinetic (qemu 7.0), Jammy (qemu 6.2) and focal (qemu 4.2) - or does the patch/commit apply cleanly?

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2023-03-07 05:56 EDT-------
Hi Frank,

after having a quick glance, for kinetic and jammy the patch should apply cleanly.

For focal, the attached patch should apply cleanly.

Revision history for this message
bugproxy (bugproxy) wrote : Proposed backport for Ubuntu Focal

------- Comment (attachment only) From <email address hidden> 2023-03-07 05:56 EDT-------

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Thanks Nico,
we just entered an SRU process with three other fixes, that needs to complete before we can prep this one.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

FYI - I have not forgotten, but the SRU cases I mentioned (a set of security fixes and bug 1921664, bug 2009048 and bug 1994002) are still ongoing for focal and jammy.

Only in Kinetic the queue is cleared already.

Changed in qemu (Ubuntu Kinetic):
status: New → In Progress
description: updated
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

As it is unlocked, this was started for Kinetic:
- PPA: https://launchpad.net/~paelzer/+archive/ubuntu/lp1999885-tod-in-pv-kinetic
- MR: https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/439453

@IBM, you'll need to help us with testing on this due to HW constraints.
Eventually you'll have to test it once accepted to -proposed for sure.
If that is easy, I'd appreciate checking the PPA as well, if not at least later on once accepted by the SRU team please.

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2023-03-24 05:03 EDT-------
(In reply to comment #13)
> As it is unlocked, this was started for Kinetic:
> - PPA:
> https://launchpad.net/~paelzer/+archive/ubuntu/lp1999885-tod-in-pv-kinetic
> - MR:
> https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/
> 439453
>
> @IBM, you'll need to help us with testing on this due to HW constraints.
> Eventually you'll have to test it once accepted to -proposed for sure.
> If that is easy, I'd appreciate checking the PPA as well, if not at least
> later on once accepted by the SRU team please.

I tried the PPA and was able to verify it fixes the issue. Thanks!

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :
Frank Heimes (fheimes)
Changed in qemu (Ubuntu Jammy):
status: New → In Progress
Changed in qemu (Ubuntu Focal):
status: New → In Progress
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

The former SRUs cleared, now uploaded to -unapproved

Revision history for this message
Steve Langasek (vorlon) wrote : Please test proposed package

Hello bugproxy, or anyone else affected,

Accepted qemu into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:7.0+dfsg-7ubuntu2.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in qemu (Ubuntu Kinetic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-kinetic
Changed in qemu (Ubuntu Jammy):
status: In Progress → Fix Committed
tags: added: verification-needed-jammy
Revision history for this message
Steve Langasek (vorlon) wrote :

Hello bugproxy, or anyone else affected,

Accepted qemu into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:6.2+dfsg-2ubuntu6.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Steve Langasek (vorlon) wrote :

Hello bugproxy, or anyone else affected,

Accepted qemu into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:4.2-3ubuntu6.26 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in qemu (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed-focal
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (qemu/1:7.0+dfsg-7ubuntu2.3)

All autopkgtests for the newly accepted qemu (1:7.0+dfsg-7ubuntu2.3) for kinetic have finished running.
The following regressions have been reported in tests triggered by the package:

ubuntu-image/2.2+22.04ubuntu3 (arm64, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/kinetic/update_excuses.html#qemu

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (qemu/1:6.2+dfsg-2ubuntu6.8)

All autopkgtests for the newly accepted qemu (1:6.2+dfsg-2ubuntu6.8) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

initramfs-tools/0.140ubuntu13.1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#qemu

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: In Progress → Fix Committed
bugproxy (bugproxy)
tags: added: targetmilestone-inin2004
removed: targetmilestone-inin---
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2023-04-19 04:10 EDT-------
Verification Result Ubuntu 22.04:
PASS

Verification Environment:
ii qemu-system-s390x 1:6.2+dfsg-2ubuntu6.8 s390x QEMU full system emulation binaries (s390x)

Linux a83lp41.lnxne.boe 5.15.0-69-generic #76-Ubuntu SMP Fri Mar 17 17:22:11 UTC 2023 s390x s390x s390x GNU/Linux

Verification Steps:
0. Start guest with SE enabled and a monitor console -> PASS
1. Verify guest timer works as expected -> PASS
~ # time sleep 1
real 0m 1.00s
user 0m 0.00s
sys 0m 0.00s
2. Stop/continue guest via QEMU monitor -> PASS, no warning message in monitor
3. Verify guest timer works as expected after stop/continue -> PASS
~ # time sleep 1
real 0m 1.00s
user 0m 0.00s
sys 0m 0.00s

Verification Result Ubuntu 20.04:
PASS

Verification Environment:
ii qemu-system-s390x 1:4.2-3ubuntu6.26 s390x QEMU full system emulation binaries (s390x)

Linux a83lp41.lnxne.boe 5.4.0-147-generic #164-Ubuntu SMP Tue Mar 21 14:22:27 UTC 2023 s390x s390x s390x GNU/Linux

Verification Steps:
0. Start guest with SE enabled and a monitor console -> PASS
1. Verify guest timer works as expected -> PASS
~ # time sleep 1
real 0m 1.00s
user 0m 0.00s
sys 0m 0.00s
2. Stop/continue guest via QEMU monitor -> PASS, no warning message in monitor
3. Verify guest timer works as expected after stop/continue -> PASS
~ # time sleep 1
real 0m 1.00s
user 0m 0.00s
sys 0m 0.00s

22.10 is still pending due to some internal issues. I hope to finish by the end of this week.

tags: added: verification-done-focal verification-done-jammy
removed: verification-needed-focal verification-needed-jammy
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2023-04-19 05:40 EDT-------
OK, good news, internal issues were resolved more quickly than expected. So here's the

Verification Result Ubuntu 22.10:
PASS

Verification Environment:
ii qemu-system-s390x 1:7.0+dfsg-7ubuntu2.3 s390x QEMU full system emulation binaries (s390x)

Linux a83lp41.lnxne.boe 5.19.0-40-generic #41-Ubuntu SMP Thu Mar 23 21:39:25 UTC 2023 s390x s390x s390x GNU/Linux

Verification Steps:
0. Start guest with SE enabled and a monitor console -> PASS
1. Verify guest timer works as expected -> PASS
~ # time sleep 1
real 0m 1.00s
user 0m 0.00s
sys 0m 0.00s
2. Stop/continue guest via QEMU monitor -> PASS, no warning message in monitor
3. Verify guest timer works as expected after stop/continue -> PASS
~ # time sleep 1
real 0m 1.00s
user 0m 0.00s
sys 0m 0.00s

tags: added: verification-done-kinetic
removed: verification-needed-kinetic
Revision history for this message
Frank Heimes (fheimes) wrote :

Many thx for the validations, Nico!

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qemu - 1:4.2-3ubuntu6.26

---------------
qemu (1:4.2-3ubuntu6.26) focal; urgency=medium

  * d/p/u/lp-1999885-s390x-tod-kvm-don-t-save-restore-the-TOD-in-PV-guest.patch:
    avoid timer issues in s390x secure execution guests (LP: #1999885)

 -- Christian Ehrhardt <email address hidden> Thu, 23 Mar 2023 08:18:28 +0100

Changed in qemu (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Chris Halse Rogers (raof) wrote : Update Released

The verification of the Stable Release Update for qemu has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qemu - 1:6.2+dfsg-2ubuntu6.8

---------------
qemu (1:6.2+dfsg-2ubuntu6.8) jammy; urgency=medium

  * d/p/u/lp-1999885-s390x-tod-kvm-don-t-save-restore-the-TOD-in-PV-guest.patch:
    avoid timer issues in s390x secure execution guests (LP: #1999885)
  * d/p/u/lp-2011832-*: fix emulation issues in mips and powerpc (LP: #2011832)

 -- Christian Ehrhardt <email address hidden> Thu, 23 Mar 2023 08:18:28 +0100

Changed in qemu (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qemu - 1:7.0+dfsg-7ubuntu2.3

---------------
qemu (1:7.0+dfsg-7ubuntu2.3) kinetic; urgency=medium

  * d/p/u/lp-1999885-s390x-tod-kvm-don-t-save-restore-the-TOD-in-PV-guest.patch:
    avoid timer issues in s390x secure execution guests (LP: #1999885)
  * d/p/u/lp-2011832-*: fix emulation issues in mips (LP: #2011832)

 -- Christian Ehrhardt <email address hidden> Thu, 23 Mar 2023 08:18:28 +0100

Changed in qemu (Ubuntu Kinetic):
status: Fix Committed → Fix Released
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.