Libvirt CPU affinity error

This guest XML would be expected to fail if the guest is running on a host that is using qemu w/o kvm acceleration:

     <vcpu placement='static' cpuset='0-1'>1</vcpu>

...as qemu does not support pinning when kvm isn't available. What's not clear to me is why the Ubuntu version would be adding this line since it sounds like master/kilo-3 is behaving correctly per the spec (only adding these lines where the user explicitly requests direct pinning of CPUs on the image or flavor [1]).

[1] http://specs.openstack.org/openstack/nova-specs/specs/juno/approved/virt-driver-cpu-pinning.html

Status changed to 'Confirmed' because the bug affects multiple users.

Yes, to test CPU pinning/NUMA with libvirt you ought to use Nested KVM.
Please report results after testing with that.

That said, some notes below.

Quoting Dan Berrange from a different review with a complete response on
*why*:

   It is fundamentally impossible to test CPU pinning with TCG (aka plain
   QEMU) because TCG only has a single thread for all virtual CPUs. As such
   there is no mechanism to pin vCPU threads with TCG. Nested KVM is thus
   the only possible option for testing any of the NUMA / CPU pinning
   stuff. Instructions for nested KVM setup on a KVM host are documented
   here

    http://docs.openstack.org/developer/nova/devref/testing/libvirt-numa.html#provisioning-a-virtual-machine-for-testing

From my testing, a Nova guest booted with a NUMA flavor, will have the
below contextual XML snippets w.r.t vCPU placement:

    . . .
      <vcpu placement='static'>4</vcpu>
      <cputune>
        <vcpupin vcpu='0' cpuset='0-3'/>
        <vcpupin vcpu='1' cpuset='0-3'/>
        <vcpupin vcpu='2' cpuset='0-3'/>
        <vcpupin vcpu='3' cpuset='0-3'/>
        <emulatorpin cpuset='0-3'/>
      </cputune>
      <numatune>
        <memory mode='strict' nodeset='0'/>
        <memnode cellid='0' mode='strict' nodeset='0'/>
      </numatune>
    . . .
      <cpu>
        <topology sockets='4' cores='1' threads='1'/>
        <numa>
          <cell id='0' cpus='0-3' memory='1048576' unit='KiB'/>
        </numa>
      </cpu>
    . . .

Here's the working example XMLs from my testing.

Libvirt XML for the guest hypervisor (also called L1), running DevStack
and will host Nova instances which are nested guests:

    https://kashyapc.fedorapeople.org/virt/openstack/nova-libvirt-numa-testing/devstack-vm-libvirt.xml

And, Nova guest XML, booted with a NUMA flavor:

    https://kashyapc.fedorapeople.org/virt/openstack/nova-libvirt-numa-testing/nova-guest-libvirt.xml

I'm not testing NUMA. I am launching a basic CirrOS image using the m1.tiny flavor, neither of which should trigger NUMA bits. In fact, adding the "hw:cpu_policy=shared" to extra_specs in the m1.tiny flavor has no impact on this issue.

Matt, you're right, allow me to correct myself below. Short: I still cannot reproduce it.

I just tested it below in a Single node DevStack with today's Nova
git with the Nova instance being QEMU emulated, but I cannot reproduce
the said failure in this bug description.

Test environment
----------------

    $ uname -r; rpm -q libvirt-daemon-kvm qemu-system-x86
    4.0.0-0.rc5.git4.1.fc22.x86_64
    libvirt-daemon-kvm-1.2.13-2.fc22.x86_64
    qemu-system-x86-2.3.0-0.2.rc1.fc22.x86_64

I'm at these commits in my All-In-One DevStack environment:

    cinder:
    commit c7ca4b95b56539dd560dc88038ab994d50c8394d
    devstack:
    commit 72bdc8c27102db3b65651ded3a9944798238a2d4
    glance:
    commit f84e49db5a455b36901b642125b5cf850f90c81d
    keystone:
    commit af568dd1afdcdc9ed7275a2824a2ca5ca50b004c
    neutron:
    commit 483de6313fab5913f9e68eb24afe65c36bd9b623
    nova:
    commit 74ca660ab688e15ccd59ddfbfcdc9e1cecdc553d
    requirements:
    commit 56ab196ad1fb0e356d3fe0ec63e744ed10104a5d

Test
----

    $ nova flavor-show 1
    +----------------------------+---------+
    | Property | Value |
    +----------------------------+---------+
    | OS-FLV-DISABLED:disabled | False |
    | OS-FLV-EXT-DATA:ephemeral | 0 |
    | disk | 1 |
    | extra_specs | {} |
    | id | 1 |
    | name | m1.tiny |
    | os-flavor-access:is_public | True |
    | ram | 512 |
    | rxtx_factor | 1.0 |
    | swap | |
    | vcpus | 1 |
    +----------------------------+---------+

Boot the instance:

    $ nova boot --config-drive false --flavor 1 \
        --key_name oskey1 --image cirros-0.3.3-x86_64-disk cirrvm1

Nova instance's guest XML attached.

That's the QEMU invocation
--------------------------

$ ps -ef | grep qemu-system-x86_64
qemu 2889 1 14 11:12 ? 00:00:11 /usr/bin/qemu-system-x86_64 -name instance-00000001 -S -machine pc-i440fx-2.3,accel=tcg,usb=off -m 512 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid cc5cae21-129b-4c63-b8e9-c642a871efdf -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=2015.1,serial=d23b2cbb-f02d-4a8e-b6dd-184a86aa8348,uuid=cc5cae21-129b-4c63-b8e9-c642a871efdf -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000001.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/home/kashyapc/src/cloud/data/nova/instances/cc5cae21-129b-4c63-b8e9-c642a871efdf/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=24,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:d1:f0:3e,bus=pci.0,addr=0x2 -chardev file,id=charserial0,path=/home/kashyapc/src/cloud/data/nova/instances/cc5cae21-129b-4c63-b8e9-c642a871efdf/console.log -d...

I'm fairly certain this is specific to the Ubuntu packages, not upstream nova or libvirt.

@Matt: Since you're farily certain that this is specific to Ubuntu, then I hope Ubuntu's Nova package maintainers will take a look. . .

Any ideas on how one could workaround this?

How to disable affinity in nova, libvirt, or qemu config's?

Some easy place in the code to hack?

Still an issue in the 2015.1~rc1-0ubuntu1~cloud0 packages. I built an environment on hosts that support KVM and successfully launched an instance, so this issue only affects QEMU. I built another environment using upstream source and successfully launched an instance using both QEMU and KVM, so I'm fairly certain this only applies to the Ubuntu packages... probably an errant patch.

Status changed to 'Confirmed' because the bug affects multiple users.

Nothing jumps out at me from the list of patches we have in the nova package - most are working around testing challenges due to offline build environments.

Raising a libvirt task to get the libvirt maintainers attention - I'll poke on irc as well.

@Matt V: I hacked an easy place in /usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py line 4720.
change
if CONF.libvirt.virt_type not in ['qemu', 'kvm']:
IN
if CONF.libvirt.virt_type not in ['kvm']:

the commit who changed the numa behavior is 945ab28.
I am not sure, does qemu without kvm has numa support?

@Matt K: Are you certain that it applies only to ubuntu packages? The changes are made upstream.

I tested in virtualbox.

My installation with packages uses the cloud archive repo that includes libvirt 1.2.12. My installation from source uses the generic 14.04 LTS repo that includes libvirt 1.2.2. Both installations use the same nova code (RC1), but the older version of libvirt doesn't exhibit this issue. Also, I'm fairly certain that QEMU doesn't support any form of NUMA.

@Matt:
in line 359 in the driver.py is the minimum libvirt version definded for which the numa code is activated.
MIN_LIBVIRT_NUMA_VERSION = (1, 2, 7)

Therfore you did not trigger the behavior with the source code installation.

I wasnt able to reproduce this on vivid.

Chuck,

What version of libvirt?

I have met the same issue:

root@openstack:~# virsh version
Compiled against library: libvirt 1.2.12
Using library: libvirt 1.2.12
Using API: QEMU 1.2.12
Running hypervisor: QEMU 2.2.0

-----------------------------------------------------------------------------
root@openstack:~# dpkg -l |grep nova
ii nova-api 1:2015.1~rc1-0ubuntu1~cloud0 all OpenStack Compute - API frontend
ii nova-cert 1:2015.1~rc1-0ubuntu1~cloud0 all OpenStack Compute - certificate management
ii nova-common 1:2015.1~rc1-0ubuntu1~cloud0 all OpenStack Compute - common files
ii nova-compute 1:2015.1~rc1-0ubuntu1~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 1:2015.1~rc1-0ubuntu1~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 1:2015.1~rc1-0ubuntu1~cloud0 all OpenStack Compute - compute node libvirt support
ii nova-conductor 1:2015.1~rc1-0ubuntu1~cloud0 all OpenStack Compute - conductor service
ii nova-consoleauth 1:2015.1~rc1-0ubuntu1~cloud0 all OpenStack Compute - Console Authenticator
ii nova-novncproxy 1:2015.1~rc1-0ubuntu1~cloud0 all OpenStack Compute - NoVNC proxy
ii nova-scheduler 1:2015.1~rc1-0ubuntu1~cloud0 all OpenStack Compute - virtual machine scheduler
ii python-nova 1:2015.1~rc1-0ubuntu1~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 1:2.22.0-0ubuntu1~cloud0 all client library for OpenStack Compute API

I also see this is still an issue with http://ubuntu-cloud.archive.canonical.com/ubuntu trusty-updates/kilo main

VERSION="14.04.1 LTS, Trusty Tahr"

# virsh version
Compiled against library: libvirt 1.2.12
Using library: libvirt 1.2.12
Using API: QEMU 1.2.12
Running hypervisor: QEMU 2.2.0

nova-compute 1:2015.1~rc1-0ubuntu1~cloud0

Status changed to 'Confirmed' because the bug affects multiple users.

Looks like a nova bug, I guess it will issue in virt_type=qemu, libvirt>=1.2.7 and compute host that supporting NUMA.

I'm a bit confused - what exactly is the bug? It is not that cpusets inside the guests to not work right? It's that the qemu guest's vcpus are not pinned to the specified cpus on the host? The host is a true hardware host? The guest is running accelerated KVM? How is it being verified that it is misbhaving - the kvm process is just not in a proper cpuset??

The hostsystem is virtualbox, and the guest system is qemu without kvm, because virtualbox doesn't support hardware acceleration.
The Problem is, that nova-compute generates an invalid xml for this combination.
The offending part is "<vcpu placement='static'>4</vcpu>".

This part is not accepted from libvirt and an error is logged.

As a result of this I am unable to test Openstack in a Vagrant Virtualbox environment.

Thanks - so the solution is for nova to only drop the placement='static' from that line, whenever it knows it is using tcg?

On the same system (QEMU only) and same version of nova (Kilo), I can launch an instance with Libvirt 1.2.2 (included with Ubuntu 14.04) but receive this error with Libvirt 1.2.12 (included with the Kilo cloud-archive repo). Either Libvirt 1.2.12 reports the wrong capabilities to nova or nova makes some sort of incorrect assumptions with it.

Can you show 'virsh capabilities' output with both packages?

The packages list of ubuntu-archive of kilo:
http://reqorts.qa.ubuntu.com/reports/ubuntu-server/cloud-archive/kilo_versions.html

But that makes no sense. If you were changing

# While earlier versions could support NUMA reporting and
# NUMA placement, not until 1.2.7 was there the ability
# to pin guest nodes to host nodes, so mandate that. Without
# this the scheduler cannot make guaranteed decisions, as the
# guest placement may not match what was requested
MIN_LIBVIRT_NUMA_VERSION = (1, 2, 7)

I could see that helping but MIN_LIBVIRT_BLOCKCOMMIT_RELATIVE_VERSION ?

@Tony
Yes. You are right. I am sorry that I make a wrong file diff. I hide my last comment, and paste a correct one.

I meet this issue too. After applied following patch, it works. I think in the ubuntu, the libvirt (1.2.12, from cloud-archive) doesn't support numa.

diff --git a/nova/virt/libvirt/driver.py b/nova/virt/libvirt/driver.py
index 98a4537..4d573e1 100644
--- a/nova/virt/libvirt/driver.py
+++ b/nova/virt/libvirt/driver.py
@@ -355,7 +355,7 @@ REQ_HYPERVISOR_DISCARD = "QEMU"
 # to pin guest nodes to host nodes, so mandate that. Without
 # this the scheduler cannot make guaranteed decisions, as the
 # guest placement may not match what was requested
-MIN_LIBVIRT_NUMA_VERSION = (1, 2, 7)
+MIN_LIBVIRT_NUMA_VERSION = (1, 2, 99)
 # While earlier versions could support hugepage backed
 # guests, not until 1.2.8 was there the ability to request
 # a particular huge page size. Without this the scheduler

It seems that nova's libvirt driver is generating an invalid domain xml. If I understand correctly specifyin a 'vcpu' node with a cpuset is invlaid in TCG *unless* you also specify emulatorpin See: https://libvirt.org/formatdomain.html#elementsCPUAllocation

This patch (which hasn't gone anywhere near upstream yet)

Forces the libvirt driver in nova to avoid generating a cpuset and there fore no longer generates the invalid domain XML.

 Next steps are to discuss my findings with upstream libvirt and nova developers to see if I'm correct of I've just fluked it.

The attachment "1439280.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

I found the root cause in my environment.

I use libvirt in LXC. And the lxc doesn't enable the cgroup w/ read and write permission. I add/change the config file with following line.

    lxc.aa_profile = lxc-container-default-with-nesting

And install `cgroup-lite` in lxc guest. The libvirt works with CPU affinity.

Using the patch above (that basically hacks out qemu specifically), nova boot work fine.
Same qemu environment as I noted in my 4-23 post.

Can we get this patch out for formal review by nova folks?

We're in discussions with the libvirt devs to work out if the fix is correct and/or exposes a libvirt bug.

Once that discussion concludes there will the a nova patch posted (and tagged for backport)

Thanks - the patch seems to make sense.

Summary of the libvirt discussion. current upstream works. the libvirt team would like to identify the libvirt fixes required and get them backported. to the maintenance releases.

With reference to:

https://launchpad.net/ubuntu/+source/libvirt ; and
http://wiki.libvirt.org/page/Maintenance_Releases

If I read those links correctly we're still going to need to fix nova and/or get the pack-ports into the appropriate ubuntu libvirt packages.

For the record. Applying this patch to the cloud-archive libvirt package should fix the problem.

http://libvirt.org/git/?p=libvirt.git;a=commit;h=a103bb105c0c189c3973311ff1826972b5bc6ad6

I was pointed at the v1.2.12-maint head in the libvirt git which contains this fix already.
http://libvirt.org/git/?p=libvirt.git;a=shortlog;h=refs/heads/v1.2.12-maint

I suggest we close the nova issue with won't fix and get the correctly backported patch into the libvirt package.

"wont-fixing" the nova side will leave it broken for quite some time until the backport has made its way into all relevant distro images. I'd prefer to add your patch into nova code as workaround for older libvirt versions.

Hello Matt, or anyone else affected,

Accepted libvirt into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/1.2.12-0ubuntu13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I can verify that installing 1.2.12-0ubuntu13 on vivid fixes the issue for me.

Please forgive my ignorance but can that package be tagged into cloud-archive once it's officially a vivid update?

@j-rosenboom-j My "fix" for nova will never be accepted upstream. I wont speak for the Ubuntu developers but I strongly suspect that they'll be unwilling to diverge from upstream. Especially as they've already shown the fix will land in vivid.

When will this get fixed for Trusty? that's where it was originally reported.

The cloud archive repository currently does not contain a fix for Kilo on Trusty.

@vanderwl: The original report was against the trusty cloud-archive repo.

If you look at: https://launchpad.net/ubuntu/+source/libvirt You can see that the version in trusty is NOT affected by this issue.

Only vivid and the could archive PPA

Matt, Tony, that's for the clarification. I'm still a bit confused as to when the cloud archive used by trusty (http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/trusty-updates/) would be updated to point to this new libvirt cloud level. (sorry for being a newbie on this).

@vanderwl No problem I'm new to cloud archive as well.

So the URL I have is: https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/kilo-staging/+index
and that shows (for me) that 4 hours ago the (kilo-staging) cloud-archive PPA got the fixed libvirt.

So in theory We're done here. Matt and I just need to verify that the new package is correct (I have no doubt it is)

I'm waiting until the package appears in the official cloud archive repository.

Status changed to 'Confirmed' because the bug affects multiple users.

This bug was fixed in the package libvirt - 1.2.12-0ubuntu13

---------------
libvirt (1.2.12-0ubuntu13) vivid-proposed; urgency=medium

  * 9038-qemu-fix-setting-of-VM-CPU-affinity-with-TCG (LP: #1439280)

 -- Serge Hallyn <email address hidden> Wed, 13 May 2015 10:48:53 -0500

The verification of the Stable Release Update for libvirt has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This should be fixed now, please re-open if it isnt.