kolla

compute container prints exception libvirtError: Failed to create controller cpu for group: No such file or directory

Bug #1473270 reported by Steven Dake
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla
Fix Released
Critical
Steven Dake
kolla liberty-2 "liberty-2"

Bug Description

    <video>^M
      <model type='cirrus' vram='16384' heads='1'/>^M
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>^M
    </video>^M
    <memballoon model='virtio'>^M
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>^M
      <stats period='10'/>^M
    </memballoon>^M
  </devices>^M
</domain>^M
^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [req-67249dfb-36d1-42f5-a952-0fefaf0f2b15 59a23b691a614f3f8218720cc9f09cc4 fd357d045b834992933316d79fb4219d - - -] [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] Instance failed to spawn^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] Traceback (most recent call last):^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2100, in _build_resources^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] yield resources^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1972, in _build_and_run_instance^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] block_device_info=block_device_info)^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2435, in spawn^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] block_device_info=block_device_info)^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4462, in _create_domain_and_network^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] xml, pause=pause, power_on=power_on)^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4392, in _create_domain^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] guest.launch(pause=pause)^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 109, in launch^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] self._encoded_xml, errors='ignore')^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 119, in __exit__^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] six.reraise(self.type_, self.value, self.tb)^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 104, in launch^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] return self._domain.createWithFlags(flags)^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] result = proxy_call(self._autowrap, f, *args, **kwargs)^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] rv = execute(f, *args, **kwargs)^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] six.reraise(c, e, tb)^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] rv = meth(*args, **kwargs)^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] File "/usr/lib64/python2.7/site-packages/libvirt.py", line 996, in createWithFlags^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)^M
2015-07-10 01:42:00.536 1 ERROR nova.compute.manager [instance: 407a3e0e-5346-43e2-91f2-27008cdff703] libvirtError: Failed to create controller cpu for group: No such file or directory^M

Revision history for this message
Steven Dake (sdake) wrote :

An internet search turns up a possible regression in libvirt:

https://bugs.launchpad.net/nova/+bug/1438226

The solution there was to check the version number of libvirt. Unfortunately CentOS (and Red Hat) cherry-pick various patches so a simple version check only works from-source built packaging. Need to investigate if CentOS version of libvirt is DOA for OpenStack.

Changed in kolla:
importance: Undecided → Critical
status: New → Triaged
assignee: nobody → Steven Dake (sdake)
milestone: none → liberty-2
Revision history for this message
Steven Dake (sdake) wrote :

Version information of libvirt in the container:
[sdake@bigiron tools]$ sudo docker exec 013 rpm -qa | grep libvirt
libvirt-daemon-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-driver-interface-1.2.8-16.el7_1.3.x86_64
libvirt-client-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-driver-qemu-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-driver-lxc-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-driver-nwfilter-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-driver-secret-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-kvm-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-config-nwfilter-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-driver-network-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-driver-nodedev-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-driver-storage-1.2.8-16.el7_1.3.x86_64

Revision history for this message
Steven Dake (sdake) wrote :

Libvirt changelog to determine if broken patches are applied to our version of libvirt:
http://ur1.ca/n2zsx -> http://paste.fedoraproject.org/242527/49541214

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (master)

Fix proposed to branch: master
Review: https://review.openstack.org/202264

Changed in kolla:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (master)

Reviewed: https://review.openstack.org/202264
Committed: https://git.openstack.org/cgit/stackforge/kolla/commit/?id=9c5e708fe247234459d153b77893b172e45760de
Submitter: Jenkins
Branch: master

commit 9c5e708fe247234459d153b77893b172e45760de
Author: Steven Dake <email address hidden>
Date: Wed Jul 15 13:15:10 2015 -0700

    Add pid_mode to libvirt

    Libvirt requires pid_mode to Host to operate on Fedora Hosts. Further
    we will need this for libvirt upgrade. I am not entirely sure why
    libvirt running with pid=host gets things working, but it definately
    has something to do with CentOS's libvirt's integration with systemd.

    Nailing down the problem, the interface to the Docker module accepts only
    None or "host". There is no way to generate a None type except with !!null
    in yml, but this does not work in the jinja2 parser. As a result, one
    solution is this conditional. A better long term solution would be for
    the Docker module to take some other argument to indicate None that can
    be expressed in a string.

    Change-Id: I54eb87e8ce8679bbf12f671527280c73e195b2e4
    Closes-Bug: #1473270

Changed in kolla:
status: In Progress → Fix Committed
Sam Yaple (s8m)
Changed in kolla:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.