rbd failure to initialize during n-cpu startup breaks n-cpu
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Confirmed
|
Wishlist
|
Unassigned |
Bug Description
CONF.libvirt.
Cinder recently addressed a similar problem in bug 1401335 by moving rbd library calls into a separate thread via eventlet.tpool. This also seems to help here.
2015-05-05 11:03:02.161 DEBUG nova.openstack.
e None] *******
log_opt_values /usr/lib/
2015-05-05 11:03:02.162 INFO nova.service [-] Starting compute node (version 2015.1)
2015-05-05 11:03:02.164 DEBUG nova.virt.
nit_events /opt/stack/
2015-05-05 11:03:02.165 DEBUG nova.virt.
_init_events /opt/stack/
2015-05-05 11:03:02.166 DEBUG nova.virt.
14406) _get_new_connection /opt/stack/
2015-05-05 11:03:02.225 INFO nova.virt.
2015-05-05 11:03:02.278 DEBUG nova.virt.
Registering for lifecycle events <nova.virt.
_new_connection /opt/stack/
2015-05-05 11:03:02.282 DEBUG nova.virt.
Registering for connection events: <nova.virt.
et_new_connection /opt/stack/
2015-05-05 11:03:02.292 INFO nova.virt.
Libvirt host capabilities <capabilities>
<host>
<uuid>
<cpu>
<
<
<
<topology sockets='4' cores='1' threads='1'/>
<feature name='hypervisor'/>
</cpu>
<power_
<
<
</power_
<migration_
<live/>
<
<
</migration
<topology>
<cells num='1'>
<cell id='0'>
<memory unit='KiB'
<cpus num='4'>
<cpu id='0' socket_id='0' core_id='0' siblings='0'/>
<cpu id='1' socket_id='1' core_id='0' siblings='1'/>
<cpu id='2' socket_id='2' core_id='0' siblings='2'/>
<cpu id='3' socket_id='3' core_id='0' siblings='3'/>
</cpus>
</cell>
</cells>
</topology>
<secmodel>
<
<doi>0</doi>
</secmodel>
<secmodel>
<
<doi>0</doi>
</secmodel>
</host>
<guest>
<os_
<arch name='i686'>
<
<
<machine canonical=
<machine maxCpus=
<machine maxCpus=
<machine canonical=
<machine maxCpus=
<machine maxCpus=
<machine maxCpus=
<machine maxCpus=
<machine maxCpus=
<machine maxCpus=
<machine maxCpus=
<machine maxCpus=
<machine maxCpus=
<machine maxCpus=
<machine maxCpus=
<machine maxCpus=
<machine maxCpus=
<machine maxCpus=
<domain type='qemu'>
</domain>
</arch>
<features>
<
<deviceboot/>
<acpi default='on' toggle='yes'/>
<apic default='on' toggle='no'/>
<pae/>
<nonpae/>
</features>
</guest>
<guest>
<os_
<arch name='x86_64'>
<
<
<machine canonical=
<machine maxCpus=
<machine maxCpus=
<machine canonical=
<machine maxCpus=
<machine maxCpus=
<machine maxCpus=
<machine maxCpus=
<machine maxCpus=
<machine maxCpus=
<machine maxCpus=
<machine maxCpus=
<machine maxCpus=
<machine maxCpus=
<machine maxCpus=
<machine maxCpus=
<machine maxCpus=
<machine maxCpus=
<domain type='qemu'>
</domain>
</arch>
<features>
<
<deviceboot/>
<acpi default='on' toggle='yes'/>
<apic default='on' toggle='no'/>
</features>
</guest>
</capabilities>
2015-05-05 11:03:02.296 DEBUG nova.virt.
e] Generated XML ('<cpu>\n <arch>x86_
<topology sockets="4" cores="1" threads="1"/>\n <feature name="hyperviso
4406) to_xml /opt/stack/
2015-05-05 11:03:02.560 INFO nova.compute.
None] Auditing locally available compute resources for node devstacktest
2015-05-05 11:08:02.606 INFO nova.virt.
irt lost: 1'
2015-05-05 11:08:02.612 ERROR oslo_messaging.
None None] Failed to publish message to topic 'conductor': heartbeat timeout
2015-05-05 11:08:02.612 TRACE oslo_messaging.
2015-05-05 11:08:02.612 TRACE oslo_messaging.
oslo_messaging/
2015-05-05 11:08:02.612 TRACE oslo_messaging.
2015-05-05 11:08:02.612 TRACE oslo_messaging.
oslo_messaging/
2015-05-05 11:08:02.612 TRACE oslo_messaging.
2015-05-05 11:08:02.612 TRACE oslo_messaging.
oslo_messaging/
2015-05-05 11:08:02.612 TRACE oslo_messaging.
conf, session, node_name)
2015-05-05 11:08:02.612 TRACE oslo_messaging.
oslo_messaging/
2015-05-05 11:08:02.612 TRACE oslo_messaging.
2015-05-05 11:08:02.612 TRACE oslo_messaging.
oslo_messaging/
2015-05-05 11:08:02.612 TRACE oslo_messaging.
ress)
2015-05-05 11:08:02.612 TRACE oslo_messaging.
2015-05-05 11:08:02.612 TRACE oslo_messaging.
qpid/messaging/
2015-05-05 11:08:02.612 TRACE oslo_messaging.
2015-05-05 11:08:02.612 TRACE oslo_messaging.
qpid/messaging/
2015-05-05 11:08:02.612 TRACE oslo_messaging.
self.error or predicate(), timeout)
2015-05-05 11:08:02.612 TRACE oslo_messaging.
qpid/messaging/
2015-05-05 11:08:02.612 TRACE oslo_messaging.
a: self.error or predicate(), timeout)
2015-05-05 11:08:02.612 TRACE oslo_messaging.
qpid/messaging/
2015-05-05 11:08:02.612 TRACE oslo_messaging.
2015-05-05 11:08:02.612 TRACE oslo_messaging.
qpid/messaging/
2015-05-05 11:08:02.612 TRACE oslo_messaging.
2015-05-05 11:08:02.612 TRACE oslo_messaging.
2015-05-05 11:08:02.612 TRACE oslo_messaging.
2015-05-05 11:08:02.653 ERROR oslo_messaging.
None None] Unable to connect to AMQP server on {'username': '', 'host': '172.16.0.10:5672', 'password':
''}: Traceback (most recent call last):
File "/usr/lib/
self.connect()
File "/usr/lib/
self._transport = trans(self.
File "/usr/lib/
self.socket = connect(host, port)
File "/usr/lib/
sock = socket.socket(af, socktype, proto)
File "/usr/lib/
notify_
File "/usr/lib/
hub.
File "/usr/lib/
self.
File "/usr/lib/
listener.
File "/usr/lib/
self.
TypeError: <lambda>() takes exactly 1 argument (0 given)
. Sleeping 1 seconds
2015-05-05 11:08:02.654 ERROR nova.openstack.
2015-05-05 11:08:02.654 TRACE nova.openstack.
2015-05-05 11:08:02.654 TRACE nova.openstack.
2015-05-05 11:08:02.654 TRACE nova.openstack.
2015-05-05 11:08:02.654 TRACE nova.openstack.
2015-05-05 11:08:02.654 TRACE nova.openstack.
2015-05-05 11:08:02.654 TRACE nova.openstack.
2015-05-05 11:08:02.654 TRACE nova.openstack.
2015-05-05 11:08:02.654 TRACE nova.openstack.
2015-05-05 11:08:02.654 TRACE nova.openstack.
2015-05-05 11:08:02.654 TRACE nova.openstack.
2015-05-05 11:08:02.654 TRACE nova.openstack.
2015-05-05 11:08:02.654 TRACE nova.openstack.
2015-05-05 11:08:02.654 TRACE nova.openstack.
2015-05-05 11:08:02.654 TRACE nova.openstack.
2015-05-05 11:08:02.654 TRACE nova.openstack.
2015-05-05 11:08:02.654 TRACE nova.openstack.
2015-05-05 11:08:02.654 TRACE nova.openstack.
2015-05-05 11:08:02.654 TRACE nova.openstack.
2015-05-05 11:08:02.654 TRACE nova.openstack.
2015-05-05 11:08:02.654 TRACE nova.openstack.
2015-05-05 11:08:02.654 TRACE nova.openstack.
Changed in nova: | |
assignee: | nobody → Eric Harney (eharney) |
status: | New → In Progress |
tags: | added: rbd |
Changed in nova: | |
assignee: | Eric Harney (eharney) → nobody |
status: | In Progress → Confirmed |
tags: |
added: ceph removed: rbd |
Related fix proposed to branch: master /review. openstack. org/180207
Review: https:/