libvirtd cannot launch VMs with SGX enabled

Hi,
thanks for the report Juro!

I'm working on libvirt 8.6 once it is released in a few more days.
Once that has entered Kinetic we can start working on an SRU for Jammy.

The list of identified patches is:
a7743c6026 qemu: process: Don't use hardcoded QOM path for cpu for probing flags
068aaa9795 qemu: process: Move call to qemuProcessRefreshCPU after cpu probe
2f498b5980 qemu: process: Move cpu flag querying after code probing cpus
f0a4951616 qemu: domain: Store 'qomPath' in qemuDomainVcpuPrivate
715846b6ea qemu: monitor: Don't hardcode QOM path of first CPU
476e864186 qemuProcessUpdateAndVerifyCPU: Refactor cleanup
3ac3b3e8fb qemuMonitorJSONGetCPUx86Data: Unexport

There are no follow up fixes to those since then we need to consider.

Hi Christian, thanks for your prompt reply. If there is anything I can do to help out, please let me know.

Thanks Juro, once I have that code ready for Jammy (as mentioned blocked by the merge of the new version to Kinetic atm) it would be great if you then could help verifying that in Jammy.

The SRU team and I will call for your help then, for now all I need is your patience with the overloaded me :-)

This bug was fixed in the package libvirt - 8.6.0-0ubuntu1

---------------
libvirt (8.6.0-0ubuntu1) kinetic; urgency=medium

  * Merge 8.0.0 from Debian unstable (LP: #1971289)
    Among many other fixes and improvements this fixes:
    - support for minor NFS versions (LP: #1980134)
    - launching VMs with SGX enabled (LP: #1982896)
    Remaining changes:
    - libvirt-uri.sh, d/rules: Automatically switch default libvirt URI
      for users via user profile (xen URI on dom0, qemu:///system otherwise)
    - Disable libssh2 support (universe dependency)
    - d/control: add libzfslinux-dev to build-deps
    - d/control: drop libvirt-lxc, vbox and xen drivers to suggest
    - debian/patches/ubuntu/ovmf_paths.patch: adjust paths to secboot.fd UEFI
      Secure Boot enabled variants of the OVMF firmware and variable store for
      the paths where we ship these files in Ubuntu.
    - Set qemu-group to kvm (for compat with older ubuntu)
    - Additional apport package-hook
    - Autostart default bridged network (As upstream does, but not Debian).
      In addition to just enabling it our solution provides:
      + do not autostart if subnet is already taken (e.g. in guests).
      + iterate some alternative subnets before giving up
    - d/p/ubuntu/Allow-libvirt-group-to-access-the-socket.patch: This is
      the group based access to libvirt functions as it was used in Ubuntu
      for quite a long time.
      + d/p/ubuntu/daemon-augeas-fix-expected.patch fix some related tests
        due to the group access change.
      + d/libvirt-daemon-system.postinst: add users in sudo to the libvirt
        group.
    - Update README.Debian with Ubuntu changes
    - d/p/ubuntu/ubuntu_machine_type.patch: accept ubuntu types as pci440fx
    - fix autopkgtests (LP 1899180)
      + d/t/control, d/t/smoke-qemu-session: fixup smoke-qemu-session by making
        vmlinuz available and accessible (Debian bug 848314)
      + d/t/control: fix smoke-qemu-session by ensuring the service will run
        installing libvirt-daemon-system
      + d/t/smoke-lxc: fix smoke-lxc by ignoring potential issues on destroy as
        long as the following undefine succeeds
      + d/t/smoke-lxc: use systemd instead of sysV to restart the service
      + d/t/control, d/t/smoke-lxc: retry service restart and skip test if
        failing; This was flaky on some release/architectures
      + d/t/smoke-lxc: retry check_domain being flaky on arm64
    - dnsmasq related enhancements
      + run dnsmasq as libvirt-dnsmasq (LP: 1743718)
      + d/libvirt-daemon-system.postinst: add libvirt-dnsmasq user and group
      + d/libvirt-daemon-system.postrm: remove libvirt-dnsmasq user and group
        on purge
      + d/p/ubuntu/dnsmasq-as-priv-user: write dnsmasq config with user
        libvirt-dnsmasq and adapt the self tests to expect that config
      + d/libvirt-daemon-system.postinst: fix old libvirt-dnsmasq users group
      + Add dnsmasq configuration to work with system wide dnsmasq-base
    - d/p/ubuntu/set-default-machine-to-ubuntu.patch: to select default
      machine type correctly with newer qemu/libvirt
    - d/p/ubuntu/lp-1861125-ubuntu-models: recognize Ubuntu models fo...

Hello Juro!

After a while, I came back, and I am about to take care of that bug.
Since I do not have an SGX system, would you mind testing PPA with the fix that I will provide?
The test plan is really complex, and one of the things that would speed up resolving that bug would be your help or taking a reproducer from another team, most likely.
Moreover, if you could test it, it would also be really valuable for us to write down a bit more detailed steps to reproduce the issue in a Ubuntu-driven guest and point out which elements from XML help to use SGX features.
Is that something you could help me with?

Hello Michal,
sorry for my belated reply. I believe I should be able to test the bug fix.

Hi Juro,

Thank you! I will provide you with the PPA by the end of this week.

Hi Juro,

The PPA is ready: https://launchpad.net/~michal-maloszewski99/+archive/ubuntu/ppa-jammy-libvirt-lp1982896
Please enter that URL and add this PPA to your jammy system as mentioned inside. (8.0.0-1ubuntu7.7~ppa1 is the correct one btw)

If you encounter any problems with that, please let me know.

Thank you in advance.

Hi Michal,
I should be able to test it within a day or so.
Thanks.

Hi Michal,
I was able to verify that using virsh/libvirt I was able to launch a VM supporting SGX.
These were my steps:

For my testing I used a NUC NUC7PJYH which is only about ~$200 abd supports SGX.
I did a fresh install of Jammy desktop, installed PPA you provided and upgraded all.
I installed various required packages libguestfs-tools, virt-manager,...

I modified /etc/libvirt/qemu.conf, editing:

cgroup_device_acl = [
   "/dev/null", "/dev/full", "/dev/zero",
   "/dev/random", "/dev/urandom",
   "/dev/ptmx", "/dev/kvm",
   "/dev/rtc","/dev/hpet",
   "/dev/sgx_enclave", "/dev/sgx_provision", "/dev/sgx_vepc"
]

user = "root"
security_driver = “none”

Started libvirtd:
$sudo systemctl start libvirtd

After that I created a CS8 VM image:

$ wget https://cloud.centos.org/centos/8-stream/x86_64/images/CentOS-Stream-GenericCloud-8-20230710.0.x86_64.qcow2

$ sudo mv CentOS-Stream-GenericCloud-8-20230710.0.x86_64.qcow2 /var/lib/libvirt/images
$ sudo virt-customize -a /var/lib/libvirt/images/CentOS-Stream-GenericCloud-8-20230710.0.x86_64.qcow2 --root-password password:123456

Edited cs8-vm-sgx.xml with the new qcow2 image name and launched the VM:

$ sudo virsh define cs8-vm-sgx1.xml
$ sudo virsh start cs8-vm-sgx1

Then I opened the running VM in virt-manager, logged into CS8 VM and verified VM supports SGX.
Screenshot attached.

Thank you for helping Michal @Juro !

Just as FYI for anyone else following this bug, this is a very crude way to enable SGX. It works, but it shouldn't be considered the nice way to do it.

Later versions of libvirt allow to do that in a much safer and expressive way including control of SGX related numa, but backporting that to Jammy is like 60+ patches and probably a stretch for the SRU policy.
So upgrading to a newer Ubuntu is the preferable way for that.

The changes requested here OTOH mostly allow the direct addition of qemu arguments as needed by Juro to get SGX activated without full libvirt support for it. Not as nice, but much less churn to consider for potential regressions that we might otherwise introduce.

Thanks @Juro, I've just seen your comment.

As Christian mentioned, I agree with what he says and there is nothing to be added to that explanation.

Now I will prepare MP for review and then continue with the SRU process :)

This statement is a bit scary for an SRU:

"""
* Since the arguments of some functions are modified it can possibly affect the API in the future.
"""

Are these functions exported in the public API, or just used internally?

Internally. Sorry, you are right. I wrote this down before I've checked it deeper and then I forgot to delete that sentence.

Done.

Hello Juro, or anyone else affected,

Accepted libvirt into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/8.0.0-1ubuntu7.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello @Juro,

As a part of the SRU Verification, can I ask you to test the libvirt package from proposed to confirm that the fix helps and resolves the issue?
It is obligatory step to test it again after the moment when the bug is 'Fix Committed' like now.

In cases where I do not have the reproducer to do that, the bug reporter "needs to" do that.

Thank you in advance.

Any update on this?

I followed the steps to enable installation of "proposed" packages:
https://wiki.ubuntu.com/Testing/EnableProposed

But I had problems with "$ sudo apt-get upgrade -s": no libvirt packages were listed.
(Running "sudo aptitude -t jammy-proposed" revealed various dependencies issues holding back libvirt packages.)

I ended up installing the libvirt packages manually:

...
$ sudo apt-get install libvirt-clients/jammy-proposed
$ sudo apt-get install libvirt-daemon-system/jammy-proposed
$ sudo apt-get install libvirt-daemon-config-network/jammy-proposed
...

This allowed me to install 8.0.0-1ubuntu7.7 versions of libvirt packages from "jammy-proposed":

$ sudo apt list --installed | grep libvirt

gir1.2-libvirt-glib-1.0/jammy,now 4.0.0-2 amd64 [installed,automatic]
libvirt-clients/jammy-proposed,now 8.0.0-1ubuntu7.7 amd64 [installed]
libvirt-daemon-config-network/jammy-proposed,jammy-proposed,now 8.0.0-1ubuntu7.7 all [installed]
libvirt-daemon-config-nwfilter/jammy-proposed,jammy-proposed,now 8.0.0-1ubuntu7.7 all [installed,automatic]
libvirt-daemon-driver-qemu/jammy-proposed,now 8.0.0-1ubuntu7.7 amd64 [installed,automatic]
libvirt-daemon-system-systemd/jammy-proposed,jammy-proposed,now 8.0.0-1ubuntu7.7 all [installed]
libvirt-daemon-system/jammy-proposed,now 8.0.0-1ubuntu7.7 amd64 [installed]
libvirt-daemon/jammy-proposed,now 8.0.0-1ubuntu7.7 amd64 [installed]
libvirt-glib-1.0-0/jammy,now 4.0.0-2 amd64 [installed,automatic]
libvirt-glib-1.0-data/jammy,jammy,now 4.0.0-2 all [installed,automatic]
libvirt0/jammy-proposed,now 8.0.0-1ubuntu7.7 amd64 [installed,automatic]
python3-libvirt/jammy,now 8.0.0-1build1 amd64 [installed,automatic]

After that I restarted and verified libvirtd:

$ sudo systemctl restart libvirtd
$ sudo systemctl status libvirtd
● libvirtd.service - Virtualization daemon
     Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2023-09-18 12:15:07 PDT; 9s ago
TriggeredBy: ● libvirtd.socket
             ● libvirtd-admin.socket
             ● libvirtd-ro.socket
       Docs: man:libvirtd(8)
             https://libvirt.org
   Main PID: 41986 (libvirtd)
      Tasks: 22 (limit: 32768)
     Memory: 19.0M
        CPU: 1.266s
     CGroup: /system.slice/libvirtd.service
             ├─13346 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_>
             ├─13347 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_>
             └─41986 /usr/sbin/libvirtd

Sep 18 12:15:07 juro-NUC7PJYH systemd[1]: Starting Virtualization daemon...
Sep 18 12:15:07 juro-NUC7PJYH systemd[1]: Started Virtualization daemon.
Sep 18 12:15:08 juro-NUC7PJYH dnsmasq[13346]: read /etc/hosts - 7 addresses
Sep 18 12:15:08 juro-NUC7PJYH dnsmasq[13346]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Sep 18 12:15:08 juro-NUC7PJYH dnsmasq-dhcp[13346]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Sep 18 12:15:08 juro-NUC7PJYH libvirtd[41986]: libvirt version: 8.0.0, package: 1ubuntu7.7 (Michal Maloszewski <email address hidden>
Sep 1...

Thank you for testing!

Thanks for verification Juro :)

This bug was fixed in the package libvirt - 8.0.0-1ubuntu7.7

---------------
libvirt (8.0.0-1ubuntu7.7) jammy; urgency=medium

  * When attempting to launch a VM with SGX enabled, there is an
    error reported that prevents VMs from being launched. Backport fix
    that fixes the main cause of that issue, which is the
    QOM_CPU_PATH macro and qom-get behavior (LP: #1982896).
    - d/p/b/qemu-monitor-json-get-cpux86-data-unexport.patch
    - d/p/b/qemu-process-update-and-verify-cpu-refactor-cleanup.patch
    - d/p/b/qemu-monitor-do-not-hardcode-qom-path-of-first-cpu.patch
    - d/p/b/qemu-domain-store-qompath-in-qemudomainvcpuprivate.patch
    - d/p/b/qemu-process-move-cpu-flag-querying-after-code-probing-cpus.patch
    - d/p/b/qemu-process-move-call-to-qemuprocessrefreshcpu-after-cpu-probe.patch
    - d/p/b/qemu-process-do-not-use-hardcoded-qom-path-for-cpu-for-probing-flags.patch

 -- Michal Maloszewski <email address hidden> Fri, 04 Aug 2023 10:42:25 +0200

The verification of the Stable Release Update for libvirt has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.