Ownership/Permissions of vhost_user sockets for openvswitch-dpdk make them unusable by libvirt/qemu/kvm

Discussion slowed down and patch series grew - still not committed.
=> https://<email address hidden>/msg57821.html

The way the discussion went this included more and more of the change from commandline to ovsdb config for dpdk.
That makes it uncertain if it will get into branch 2.5

Had a discussion with James Page today.
He will look into it (backporting, hack, other solution) in his work regarding charming that stuff.
Big thanks for that!

We agreed that will discuss (or be happy about the fix) again in approx 10 days from now.

Looking forward for a proper fix to this problem! Very annoying... :-P

OVS+DPDK on Ubuntu 16.04 looks impressive!

Adding libvirt task as well as one solution to this is for libvirt to fix the permissions on the socket at the point of vm creation, as it does for disk permissions...

FYI - Solution for DPDK has passed tests now.
I sent it to DPDK for review and to get their general opinion on it, but we might carry it to be able to work with openvswitch-2.5 as is anyway.

To be toally fixed this will need an DPDK upload (soon in the queue) to work.

We will also update openvswitch e.g. the examples in /etc/defaults/openvswitch that I'll start to prepare now - this is to fix the other portion of the bug in openvswitch.
But this portion will likely be uploaded together with the 2.5.1 point release which fixes some serious openvswitch-dpdk issues anyway.

This change is supposed to be uploaded together with the merge of the next point release as requested in bug 1573091.
This will fix several critical issues, and without those in place this update of the "how to config" isn't too important yet.

So no need to sponsor this now (in case the bot auto assigns sponsors later).

The attachment "Debdiff for the recommended changes to the examples in /etc/defaults/openvswitch" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

--vhost-perm 0664 would make even more sense.

Xenial is released, so we are back in SRU mode.
Therefore I add the matching SRU Template for the upload of 2.2.0ubuntu8 which is in the unapproved queue atm.

[Impact]

 * Openvswitch-DPDK not usable without unacceptable workarounds
 * This provides a stable way to get it set up which will be picked up by our packaged openvswitch
 * We also drive a long term solution upstream at ovs/dpdk to drop the delta somewhen in the future

[Test Case]

 * Creating vhost_user sockets with ovs-dpdk always ends up as root:root

[Regression Potential]

 * no change when the dpdk EAL parameters are not set, so low regression potential
 * passed ADT tests on i368/amd64/amd64-lowmem and our full CI (https://code.launchpad.net/~ubuntu-server/ubuntu/+source/dpdk-testing/+git/dpdk-testing)

Christian, please upload the packge to yakkety too, otherwise this cannot progress to xenial-updates.

Hello ChristianEhrhardt, or anyone else affected,

Accepted dpdk into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dpdk/2.2.0-0ubuntu8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

FYI - Verified in Proposed.

Next I need to prep some Y tests to reasonably request an upload to Yakkety to allow migration as Martin indicated.

Thanks Martin for making me aware of the Y upload needed,
I did ensure that it passed sbuild and adt in Y environment, these passed, so now the debdiff for Y in this attachment.

This bug was fixed in the package dpdk - 2.2.0-0ubuntu9

---------------
dpdk (2.2.0-0ubuntu9) yakkety; urgency=medium

  * d/p/ubuntu-backport-[36-37] fix virtio issues (LP: #1570195):
    - don't let DPDK initialize virtio devices still in use by the kernel
    - this avoids conflicts between kernel and dpdk usage of those devices
    - an admin now has to unbind/bind devices as on physical hardware
    - this is in the dpdk 16.04 release and delta can then be dropped
    - d/dpdk-doc.README.Debian update for changes in virtio-pci handling
    - d/dpdk.interfaces update for changes in virtio-pci handling
  * d/p/ubuntu-backport-38... fix for memory leak (LP: #1570466):
    - call vhost_destroy_device on removing vhost user ports to fix memory leak
    - this likely is in the dpdk 16.07 release and delta can then be dropped
  * d/p/ubuntu-fix-vhost-user-socket-permission.patch fox (LP: #1546565):
    - when vhost_user sockets are created they are owner:group of the process
    - the DPDK api to create those has no way to specify owner:group
    - to fix that without breaking the API and potential workaround code in
      consumers of the library like openvswitch 2.6 for example. This patch
      adds an EAL commandline option to specify user:group created vhost_user
      sockets should have.

 -- Christian Ehrhardt <email address hidden> Wed, 27 Apr 2016 07:52:48 -0500

Hey guys,

Sorry to ask this here but, how to use the fix ?

I have the proposed repository enabled, the latest DPDK that contains the fix is installed (apt full-upgrade on top of Proposed) but, look:

---
virsh start ubuntu16.04-1
error: Failed to start domain ubuntu16.04-1
error: internal error: process exited while connecting to monitor: 2016-04-29T23:22:57.647708Z qemu-system-x86_64: -chardev socket,id=charnet2,path=/var/run/openvswitch/vhost-user1: Failed to connect socket: Permission denied

---

ll /var/run/openvswitch/vhost-user*
srwxr-xr-x 1 root root 0 Apr 29 23:20 /var/run/openvswitch/vhost-user1=
srwxr-xr-x 1 root root 0 Apr 29 23:20 /var/run/openvswitch/vhost-user2=

---

Thanks!
Thiago

Never mind... I'm reading the "ubuntu-fix-vhost-user-socket-permission.patch" file:

---
+* ``--vhost-owner``
+
+ When creating vhost_user sockets change owner and group to the specified value.
+ This can be given as ``user:group``, but also only ``user`` or ``:group`` are supported.
+
+ Examples::
+
+ --vhost-owner 'libvirt-qemu:kvm'
+ --vhost-owner 'libvirt-qemu'
+ --vhost-owner ':kvm'
+
+* ``--vhost-perm``
+
+ When creating vhost_user sockets set them up with these permissions.
+
+ For example::
+
+ --vhost-perm '0664'
---

I'll try it now...

Cheers!
Thiago

This bug was fixed in the package dpdk - 2.2.0-0ubuntu8

---------------
dpdk (2.2.0-0ubuntu8) xenial; urgency=medium

  * d/p/ubuntu-backport-[36-37] fix virtio issues (LP: #1570195):
    - don't let DPDK initialize virtio devices still in use by the kernel
    - this avoids conflicts between kernel and dpdk usage of those devices
    - an admin now has to unbind/bind devices as on physical hardware
    - this is in the dpdk 16.04 release and delta can then be dropped
    - d/dpdk-doc.README.Debian update for changes in virtio-pci handling
    - d/dpdk.interfaces update for changes in virtio-pci handling
  * d/p/ubuntu-backport-38... fix for memory leak (LP: #1570466):
    - call vhost_destroy_device on removing vhost user ports to fix memory leak
    - this likely is in the dpdk 16.07 release and delta can then be dropped
  * d/p/ubuntu-fix-vhost-user-socket-permission.patch fox (LP: #1546565):
    - when vhost_user sockets are created they are owner:group of the process
    - the DPDK api to create those has no way to specify owner:group
    - to fix that without breaking the API and potential workaround code in
      consumers of the library like openvswitch 2.6 for example. This patch
      adds an EAL commandline option to specify user:group created vhost_user
      sockets should have.

 -- Christian Ehrhardt <email address hidden> Mon, 25 Apr 2016 11:42:40 +0200

The verification of the Stable Release Update for dpdk has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Fix proposed to branch: master
Review: https://review.openstack.org/314472

Reviewed: https://review.openstack.org/314472
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-openvswitch/commit/?id=4f6e2ca2512e298faf17b1db532625132623a628
Submitter: Jenkins
Branch: master

commit 4f6e2ca2512e298faf17b1db532625132623a628
Author: James Page <email address hidden>
Date: Tue May 10 10:16:06 2016 +0100

    Set correct permissions for vhostuser sockets

    The latest updates to DPDK in 16.04 and above introduce two new
    parameters for DPDK initialization which avoid the need to run
    qemu processes with vhostuser sockets as root.

    Use these options to ensure that sockets are created with the
    correct ownership and permissions for OpenStack/KVM.

    Change-Id: I04bbd514d1bdb9b3249ed69e8d64eb66d9839944
    Closes-Bug: 1546565

Please send this patch to Debian as well, diverging config files in /etc sound bad in the long run.

Hi,
trying to clarify the complex situation regarding submitting this to Debian.

1. DPDK is not yet available in Debian. The to be packaged version is based on Ubuntu packaging and will have this fix.
2. Openvswitch as in Debian atm is not yet DPDK compatible (at 2.3 and would need to be >=2.5). Thereby the patch makes no sense for Debian (yet)

Once these are available in Debian it makes sense to submit to reduce delta.
Well actually we don't derive openvswitch from debian so it is no debian delta - still it would be nice to stay in sync.

I'm patch pilot today and a little confused. Can you check my summary of the situation?

The reason this is on the sponsorship report is to get the patch in comment #6 uploaded to yakkety? Martin asked for it to be sent to Debian, but as there is no dpdk in debian yet, there is no support for it in openvswitch in debian either, so there is nowhere to send the patch. As such the patch seems reasonable to me. Hopefully when Debian does get support from dpdk they won't do anything gratuitously different to what Ubuntu did...

Hi Michael,
this seems to confuse people regularly, so instead of just answering questions as last time I went over the whole bug again to summarize. Thanks for being persistent pitti and mwhudson!

TL;DR
please unsubscribe sponsors and SRU Verification

Summary:

Upstream DPDK & Openvswitch (and libvirt):
This is a back and forth discussion who should fix it and who not. It is essentially looping around with no project accepting a fix yet and leaving the overall solution (OVS+DPDK on KVM Guests) unusable for end users.
That is why we carry the delta for now - DPDK to provide the option, Openvswitch to show an example how to exploit the option.

DPDK (fix to make option available):
- fixed in Yakkety 2.2.0-0ubuntu9
- fixed in Xenial 2.2.0-0ubuntu8
- Debian (I'm co-leading the Debian effort, the fix is in there already)

neutron-openvswitch Juju Charms Collection (exploit this from charms):
- already fixed in 4f6e2ca2 by James Page

Openvswitch (just add example to default config):
- the debdiff of #6 should have gone to Xenial
- since we now have yakkety I'll add another debdiff for yakkety
- Debian needs to get a recent Openvswitch first and the DPDK with the patch before we can push
- given that:
  - upstream hasn't decided on a solution yet
  - it is only the example that we wanted to be extended
  - it is covered in https://help.ubuntu.com/16.04/serverguide/DPDK.html#dpdk-openvswitch
  => I think we can drop this for now.