libvirt fails with failure to open mount namespace

So, investigated this into an apparmor difference between 4.17 and 4.18. This is due to commit 338d0be437ef10e247a35aed83dbab182cf406a2 ("apparmor: fix ptrace read check").

libvirtd here is using only trace, and not read. The patch below for libvirtd apparmor policy fixes it for me.

--- /etc/apparmor.d/usr.sbin.libvirtd 2018-08-23 14:52:04.574252908 -0300
+++ ../usr.sbin.libvirtd 2018-08-23 14:51:46.773728841 -0300
@@ -50,10 +50,10 @@
   # for --p2p migrations
   unix (send, receive) type=stream addr=none peer=(label=unconfined addr=none),

- ptrace (trace) peer=unconfined,
- ptrace (trace) peer=/usr/sbin/libvirtd,
- ptrace (trace) peer=/usr/sbin/dnsmasq,
- ptrace (trace) peer=libvirt-*,
+ ptrace (read,trace) peer=unconfined,
+ ptrace (read,trace) peer=/usr/sbin/libvirtd,
+ ptrace (read,trace) peer=/usr/sbin/dnsmasq,
+ ptrace (read,trace) peer=libvirt-*,

   signal (send) peer=/usr/sbin/dnsmasq,
   signal (read, send) peer=libvirt-*,

ago 23 14:40:18 calabresa audit[5997]: AVC apparmor="DENIED" operation="ptrace" profile="/usr/sbin/libvirtd" pid=5997 comm="libvirtd" requested_mask="read" denied_mask="read" peer="unconfined"
ago 22 18:00:12 calabresa audit[4409]: AVC apparmor="DENIED" operation="ptrace" profile="/usr/sbin/libvirtd" pid=4409 comm="libvirtd" requested_mask="read" denied_mask="read" peer="libvirt-14e92a75-7668-4b97-8f92-322fc1b9c78a"
ago 23 09:18:06 calabresa audit[2507]: AVC apparmor="DENIED" operation="ptrace" profile="/usr/sbin/libvirtd" pid=2507 comm="libvirtd" requested_mask="read" denied_mask="read" peer="unconfined"

Here are some messages with the apparmor denial.

Tested, confirmed wrapped in a libvirt upstream patch and submitted at https://www.redhat.com/archives/libvir-list/2018-August/msg01532.html

Lets give the people a few days to read and ack as well then we can push it to cosmic.

It came to my mind that due to HWE-Kernels we eventually also want this in Bionic.
So lets add a task right away to not forget.

Upstream accepted: https://libvirt.org/git/?p=libvirt.git;a=commit;h=8741b9435108b1f0d87670e44e1ed75f806b7791

Prepared in PPA: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3381

Tested against Kernel 4.18 from mainline builds:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.18/linux-image-unsigned-4.18.0-041800-generic_4.18.0-041800.201808122131_amd64.deb

For the latter SRU I summarized the testing procedure:
Note: This can but does not have to be tested in nested virtualization (Since we need to change the kernel it is probably the preferred way to do it in nested, so this is what I describe)
1. Get a KVM guest with the Release to test (Cosmic as an example) and give it a size to be able to host a nested guest
  $ sudo apt install uvtool-libvirt
  $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 release=cosmic label=daily
  $ uvt-kvm create --cpu 8 --memory 4096 --password ubuntu c-nested-kvm-4.18 arch=amd64 release=cosmic label=daily
  $ uvt-kvm ssh c-nested-kvm-4.18
2. In the guest set up another guest
  $ uvt-kvm ssh c-nested-kvm-4.18
  $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 release=cosmic label=daily
  $ uvt-kvm create --password ubuntu c-nested-guest arch=amd64 release=cosmic label=daily
  $ sudo apt install uvtool-libvirt
3. The above will work, now get a kernel >=4.18 e.g. the mainline kernels at
   https://wiki.ubuntu.com/Kernel/MainlineBuilds
   $ wget (link of the day)
   $ sudo dpkg -i *.deb
4. reboot into the 4.18 kernel
   $ sudo reboot
5. Start the guest again
   $ virsh start c-nested-guest

Without fix this will fail like:
  error: Failed to start domain c-nested-guest
  error: internal error: child reported: Kernel does not provide mount namespace: Permission denied

And an associated Dmesg/Apparmor entry:
[ 519.529760] audit: type=1400 audit(1535364820.976:27): apparmor="DENIED" operation="ptrace" profile="/usr/sbin/libvirtd" pid=1692 comm="libvirtd" requested_mask="read" denied_mask="read" peer="libvirt-4ae5dbbe-641c-4934-ae7c-956bcc8b9e2f"

6. In the guest update from PPA/Proposed that you want to test to bring in the libvirt package with the updated rules (Example with the PPA).
  $ sudo add-apt-repository ppa:ci-train-ppa-service/3381
  $ sudo apt upgrade

7. Start the guest again and it will now work
  $ virsh start c-nested-guest

Tested from PPA, also passed through some more checks and all look fine - but that was expected given the change being "just" opening up apparmor a little bit.

Uploading to Cosmic ...

FYI cosmic is slightly stalled by glibc migration, other than that it looks good to me at the moment.

PPA for the Bionic SRU at https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3386
Note: This not only waits for Cosmic to complete, but also for the former SRU to complete as well.

This bug was fixed in the package libvirt - 4.6.0-2ubuntu3

---------------
libvirt (4.6.0-2ubuntu3) cosmic; urgency=medium

  * d/p/ubuntu-aa/lp-1788603-fix-ptrace-rules-with-kernel-4.18.patch:
    avoid issues with newer kernels >=4.18 (LP: #1788603)

 -- Christian Ehrhardt <email address hidden> Mon, 27 Aug 2018 10:57:57 +0200

Ok, glibc was unblocked and we can take a look at pushing this for Bionic as well.

Had another set of regression tests as I bundled another fix.
All good still.

Pushing for the SRU teams consideration ...

Hello Thadeu, or anyone else affected,

Accepted libvirt into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/4.0.0-1ubuntu8.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Started a guest with Bionic and Kernel 4.18 - working due to the updated apparmor profile.
We already had general regression tests on the identical PPA content, not redoing the same.

Setting verified.

Status changed to 'Confirmed' because the bug affects multiple users.

The verification of the Stable Release Update for libvirt has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package libvirt - 4.0.0-1ubuntu8.5

---------------
libvirt (4.0.0-1ubuntu8.5) bionic; urgency=medium

  * d/p/ubuntu-aa/lp-1788603-fix-ptrace-rules-with-kernel-4.18.patch:
    avoid issues with newer kernels >=4.18 (LP: #1788603)
  * d/p/ubuntu/lp-1789659-don-t-check-for-parallel-iteration-in-hash.patch:
    remove broken and redundant check for parallel iteration in hash functions
    (LP: #1789659)

 -- Christian Ehrhardt <email address hidden> Tue, 28 Aug 2018 07:26:19 +0200