[22.04 FEAT] Enhanced Interpretation for PCI Functions - qemu part

Please specify the planned qemu target release this is going to become upstream accepted.
Will it come with 4.2?
Changing to Incomplete for now.

Hi,
did this get into qemu 4.2 or git already?

------- Comment From <email address hidden> 2020-01-28 04:12 EDT-------
Will not make if for 20.04 -> moved to 20.10

Hi, did this get into qemu 5.0?
or into git to be eventually in 5.1?

------- Comment From <email address hidden> 2020-08-27 02:26 EDT-------
Feature request moved to 21.04. Will not make it in time for 20.10

------- Comment From <email address hidden> 2021-02-23 03:51 EDT-------
Feature will not make it into 21.04, Moved to 21.10

Hi,
I didn't see this mentioned in [1] nor an obvious commit.
Is this in 6.0 or at least the master branch by now?

[1]: https://wiki.qemu.org/ChangeLog/6.0#s390

------- Comment From <email address hidden> 2021-08-31 12:20 EDT-------
Feature did not not make it into Impish / 21.10, therefore moving to 22.04
Target Milestone:->22.04

Is there an update on this feature upstream in 6.2 or to backport now?
https://wiki.qemu.org/ChangeLog/6.2#s390x does not list it AFAICS.

------- Comment From <email address hidden> 2022-07-31 21:59 EDT-------
Latest upstream version of the QEMU series can be found here: https://<email address hidden>/

------- Comment From <email address hidden> 2022-09-01 08:19 EDT-------
The code didn't make it into QEMU 7.1 but should be available with the next release.
Changing target version to: QEMU 7.2

Hiho,
so far the series that was referenced didn't land afaics.
The latest s390x/pci code is still
  ac6aa30ac4 s390x/pci: add supported DT information to clp response
From v7.0

Given that the series isn't trivial, not yet landed and we are well in feature freeze I'm marking this for Ubuntu 23.04 which should match the qemu-7.2 that you are now targetting by your last update.

------- Comment From <email address hidden> 2022-11-29 22:09 EDT-------
Series landed in upstream master QEMU v7.2

Relevant commits (newest first) are:

59d1ce4439 s390x/s390-virtio-ccw: add zpcii-disable machine property
9ee8f7e46a s390x/pci: reflect proper maxstbl for groups of interpreted devices
30dcf4f7fd s390x/pci: let intercept devices have separate PCI groups
d0bc7091c2 s390x/pci: enable adapter event notification for interpreted devices
15d0e7942d s390x/pci: don't fence interpreted devices without MSI-X
dd1d5fd968 s390x/pci: enable for load/store interpretation
21fa15298d s390x/pci: add routine to get host function handle from CLP info
d525f73f91 Update linux headers to v6.0-rc4
28d01b1d69 configure: Add -Wno-gnu-variable-sized-type-not-at-end

Hi,
thanks for the update. I'm merging qemu 7.2 now, I'll tag this as closed once it is fully complete.

Meanwhile 7.2 landed in lunar-proposed:
qemu | 1:7.2+dfsg-3ubuntu1 | lunar-proposed
hence updating ticket status to Fix Committed.

This bug was fixed in the package qemu - 1:7.2+dfsg-4ubuntu1

---------------
qemu (1:7.2+dfsg-4ubuntu1) lunar; urgency=medium

  * Merge with Debian unstable (LP: #1993438), among many other fixes
    this resolvs these bugs:
    (LP: #1957924) - support for querying stats,
    (LP: #1853307) - Enhanced Interpretation for PCI Functions (s390x)
    (LP: #1959966) - guest dump encryption with customer keys (s390x)
    (LP: #1999885) - pv: don't allow userspace to set the clock under PV
    (LP: #1957924) - add filtering of statistics by target vCPU
    remaining changes:
    - qemu-kvm to systemd unit
      - d/qemu-kvm-init: script for QEMU KVM preparation modules, ksm,
        hugepages and architecture specifics
      - d/qemu-system-common.qemu-kvm.service: systemd unit to call
        qemu-kvm-init
      - d/qemu-system-common.install: install helper script
      - d/qemu-system-common.qemu-kvm.default: defaults for
        /etc/default/qemu-kvm
      - d/rules: call dh_installinit and dh_installsystemd for qemu-kvm
    - Distribution specific machine type
      (LP: 1304107 1621042 1776189 1761372 1761372 1776189)
      - d/p/ubuntu/define-ubuntu-machine-types.patch: define distro machine
        types containing release versioned machine attributes
      - d/qemu-system-x86.NEWS Info on fixed machine type defintions
        for host-phys-bits=true
      - Add an info about -hpb machine type in debian/qemu-system-x86.NEWS
      - ubuntu-q35 alias added to auto-select the most recent q35 ubuntu type
    - Enable nesting by default
      - d/p/ubuntu/enable-svm-by-default.patch: Enable nested svm by default
        in qemu64 on amd
        [ No more strictly needed, but required for backward compatibility ]
    - tolerate ipxe size change on migrations to >=18.04 (LP: 1713490)
      - d/p/ubuntu/pre-bionic-256k-ipxe-efi-roms.patch: old machine types
        reference 256k path
      - d/control-in: depend on ipxe-qemu-256k-compat-efi-roms to be able to
        handle incoming migrations from former releases.
    - d/qemu-system-x86.README.Debian: add info about updated nesting changes
    - Ease the use of module retention on upgrades (LP 1913421)
      - debian/qemu-block-extra.postinst: enable mount unit on install/upgrade
    - d/control-in: switch qemu-system-x86-xen to qemu-system-xen as this
      landed in Debian but under a different name.
    - Remaining GCC-12 FTBFS (LP 1988710 + LP 1921664)
      + d/p/u/qboot-Disable-LTO-for-ELF-binary-build-step.patch:
        fix qboot FTBFS with LTO
  * Dropped Changes [now part of upstream v7.2.0]
    - d/p/u/lp1994002-migration-Read-state-once.patch: Fix for libvirt
      error 'migration was active, but no RAM info was set' (LP 1994002)
    - d/p/u/ebpf-replace-deprecated-bpf_program__set_socket_filt.patch:
      Fix FTBFS with libbpf 1.0.1-2.
      + Header updates that were added as part of the libbpf fixes
        but not mentioned in changelog
    - d/p/u/lp-1981339-*: fix s390x system emulation (LP 1981339)
    - Fix I/O stalls when using NVMe storage (LP 1970737).
      + d/p/lp1970737-linux-aio-*.patch: Fix unbalanced plugged counter
        in laio_io_unplug.
    - SECURITY UPDATE...

Since we have this now in lunar, we could go ahead if we have the patches for qemu (kinetic and jammy).

Since we have this not in lunar, we could go ahead if we have the patches for qemu (kinetic and jammy).

------- Comment From <email address hidden> 2023-05-17 10:26 EDT-------
Sorry for the belated response, I missed the request for this backport.

For backporting to kinetic, the following is required and can be cherry-picked (except where noted a proposed backport has been provided):

e4082063e4 linux-headers: Update to v5.18-rc6
92e879505f linux-headers: Add vduse.h
<backport of 105bb7cdbe kvm: add support for boolean statistics>
<backport of 67f7e426e5 hw/i386: pass RNG seed via setup_data entry>
28d01b1d69 configure: Add -Wno-gnu-variable-sized-type-not-at-end
d525f73f91 Update linux headers to v6.0-rc4
21fa15298d s390x/pci: add routine to get host function handle from CLP info
dd1d5fd968 s390x/pci: enable for load/store interpretation
15d0e7942d s390x/pci: don't fence interpreted devices without MSI-X
d0bc7091c2 s390x/pci: enable adapter event notification for interpreted devices
30dcf4f7fd s390x/pci: let intercept devices have separate PCI groups
9ee8f7e46a s390x/pci: reflect proper maxstbl for groups of interpreted devices
4a8d21ba50 s390x/pci: RPCIT second pass when mappings exhausted
df202e3ff3 s390x/pci: shrink DMA aperture to be bound by vfio DMA limit
03451953c7 s390x/pci: reset ISM passthrough devices on shutdown and system reset

For backporting to jammy, the following is required and can be cherry-picked (except where noted a proposed backport has been provided):

a4663f1a55 virtio-gpu: do not byteswap padding
43709a0ca3 linux-headers: update to 5.16-rc1
b91a0fa70c update-linux-headers: Add asm-riscv/kvm.h
ef17dd6a8e linux-headers: Update headers to v5.17-rc1
1ea5208feb linux-headers: include missing changes from 5.17
e4082063e4 linux-headers: Update to v5.18-rc6
92e879505f linux-headers: Add vduse.h
<backport of 105bb7cdbe kvm: add support for boolean statistics>
<backport of 67f7e426e5 hw/i386: pass RNG seed via setup_data entry>
b2892a2b9d s390x/pci: use a reserved ID for the default PCI group
ac6aa30ac4 s390x/pci: add supported DT information to clp response
28d01b1d69 configure: Add -Wno-gnu-variable-sized-type-not-at-end
d525f73f91 Update linux headers to v6.0-rc4
21fa15298d s390x/pci: add routine to get host function handle from CLP info
dd1d5fd968 s390x/pci: enable for load/store interpretation
15d0e7942d s390x/pci: don't fence interpreted devices without MSI-X
d0bc7091c2 s390x/pci: enable adapter event notification for interpreted devices
30dcf4f7fd s390x/pci: let intercept devices have separate PCI groups
9ee8f7e46a s390x/pci: reflect proper maxstbl for groups of interpreted devices
4a8d21ba50 s390x/pci: RPCIT second pass when mappings exhausted
df202e3ff3 s390x/pci: shrink DMA aperture to be bound by vfio DMA limit
03451953c7 s390x/pci: reset ISM passthrough devices on shutdown and system reset

Please note: The 2 backport patches provided can be used for both kinetic and jammy (same diffs)

------- Comment (attachment only) From <email address hidden> 2023-05-17 10:23 EDT-------

------- Comment (attachment only) From <email address hidden> 2023-05-17 10:24 EDT-------

Discussing with Fheimes, we decided to ask to skip the double effort for kinetic.
Planning to go to Jammy only - eventually with all build & test almost as much time will pass that Kinetic won't matter.

> The test and verification steps will be performed by our partners at IBM.

Please provide the exact test and verification steps that are planned to be carried out.

------- Comment From <email address hidden> 2023-07-18 09:56 EDT-------
Testing will consist of the following (all on s390):

Hardware used: z14 or greater LPAR, PCI-attached devices (RoCE VFs, ISM devices, NVMe drive)

Setup: Both the kernel and QEMU features are needed for the feature to function (an upstream QEMU can be used to verify the kernel early), and the facility is only avaialble on z14 or newer. When any of those pieces is missing, the interpretation facility will not be used.
When both the kernel and QEMU features are included in their respective packages, and running in an LPAR on a z14 or newer machine, this feature will be enabled automatically. Existing supported devices should behave as before with no changes required by an end-user (e.g. no changes to libvirt domain definitions) -- but will now make use of the interpretation facility.
Additionally, ISM devices will now be eligible for vfio-pci passthrough (where before QEMU would exit on error if attempting to provide an ISM device for vfio-pci passthrough, preventing the guest from starting)

Testing will include the following scenarios, repeated each for RoCE, ISM and NVMe:

1) Testing of basic device passthrough (create a VM with a vfio-pci device as part of the libvirt domain definition, passing through a RoCE VF, an ISM device, or an NVMe drive. Verify that the device is available in the guest and functioning)
2) Testing of device hotplug/unplug (create a VM with a vfio-pci device, virsh detach-device to remove the device from the running guest, verify the device is removed from the guest, then virsh attach-device to hotplug the device to the guest again, verify the device functions in the guest)
3) Host power off testing: Power off the device from the host, verify that the device is unplugged from the guest as part of the poweroff
4) Guest power off testing: Power off the device from within the guest, verify that the device is unusuable in the guest, power the device back on within the guest and verify that the device is once again usable.
5) Guest reboot testing: (create a VM with a vfio-pci device, verify the device is in working condition, reboot the guest, verify that the device is still usable after reboot)

Testing will include the following scenarios specifically for ISM devices:

1) Testing of SMC-D v1 fallback: Using 2 ISM devices on the same VCHID that share a PNETID, create 2 guests and pass one ISM device via vfio-pci device to each guest. Establish TCP connectivity between the 2 guests using the libvirt default network, and then use smc_run (https://manpages.ubuntu.com/manpages/jammy/man8/smc_run.8.html) to run an iperf workload between the 2 guests (will include both short workloads and longer-running workloads). Verify that SMC-D transfer was used between the guests instead of TCP via 'smcd stats' (https://manpages.ubuntu.com/manpages/jammy/man8/smcd.8.html)

2) Testing of SMC-D v2: Same as above, but using 2 ISM devices on the same VCHID that have no PNETID specified

Testing will include the following scenarios specifically for RoCE devices:

1) Ping testing: Using 2 RoCE VFs that share a common network, create 2 guests and pass one RoCE device...

Many thx @mjrosato - I've moved your test plan to the Bug description and made it part of the SRU justification.

The qemu package will be made available (first of all via -proposed) when this SRU got approved and accepted.

Thanks mjrosato and Frank. I marked the Jammy task as In Progress again.

Let me cross reference (as fyi) the kernel verification result:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1853306/comments/31

Hello bugproxy, or anyone else affected,

Accepted qemu into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:6.2+dfsg-2ubuntu6.13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted qemu (1:6.2+dfsg-2ubuntu6.13) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

ganeti/3.0.2-1ubuntu1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#qemu

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

------- Comment From <email address hidden> 2023-08-10 04:49 EDT-------
Sorry for the delay. Due to vacation, the verification cannot be done prior to next week / Aug. 16

------- Comment From <email address hidden> 2023-08-16 14:18 EDT-------
Sorry for the delay here.

Verification was successful, with all test scenarios passing. All tests were run using the kernel in jammy-proposed on the host as well as in the KVM guest + the qemu from jammy-propsed (1:6.2+dfsg-2ubuntu6.13). The testing performed was described in an earlier comment and was essentially a repeat of the testing used to verify the kernel feature (LP#1853306) - but now with both a kernel and qemu from jammy-proposed. All testing was done on s390.

As far as test results, same as described in the associated kernel feature (LP#1853306) -- NVMe and RoCE testing went exactly as expected. For ISM testing, I ran into the same kernel ISM driver warning as described during the kernel verification and employed the same workarounds to successfully complete testing.

------- Comment From <email address hidden> 2023-08-16 16:59 EDT-------
Hi Matt, thanks for the immediate verification right after your return from vacation!

Many thanks for the verification!

> Aside from the extensive test described above, we will also run the qemu-migration-test
> (<https://code.launchpad.net/~ubuntu-server/ubuntu/+source/qemu-migration-test/+git/qemu-migration-test>)
> to guarantee that non-s390x architectures aren't affected by the set of patches being backported.

Sergio, do you have these non-s390x results?

Unfortunately we're having problems with libvirt on Mantic which are making it impossible to run qemu-migration-test. I don't have the results yet.

Since the qemu-migration-test run is still missing, I'm changing the tag back to verification-needed.

The verification of this bug is somewhat blocked by bug #2033325. I think I can workaround the testing scripts to exclude Mantic from the mix. I'll try that tomorrow.

I was finally able to run the test scripts, and here are the results for Jammy:

prep (x86_64) : Pass 20 F/S/N 0/0/0 - RC 0 (17 min 41725 lin)
migrate (x86_64) : Pass 280 F/S/N 0/0/0 - RC 0 (65 min 226281 lin)
cross (x86_64) : Pass 24 F/S/N 0/0/0 - RC 0 (46 min 41522 lin)
misc (x86_64) : Pass 73 F/S/N 0/0/0 - RC 0 (30 min 43375 lin)

As can be seen, all tests completed successfully for all stages. Therefore, I consider the verification for Jammy as being completed. Tagging the bug accordingly.

This bug was fixed in the package qemu - 1:6.2+dfsg-2ubuntu6.13

---------------
qemu (1:6.2+dfsg-2ubuntu6.13) jammy; urgency=medium

  * d/p/u/lp-1853307-*.patch: Backport patches to implement Enhanced
    Interpretation for PCI Functions (s390x). (LP: #1853307)

 -- Sergio Durigan Junior <email address hidden> Wed, 05 Jul 2023 10:47:05 -0400

The verification of the Stable Release Update for qemu has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

------- Comment From <email address hidden> 2023-09-01 10:28 EDT-------
With the release to jammy, we can finally close this bug.

Thanks everyone for your hard work that went into this item!

==> Changing the status to "CLOSED"