I believe I have reproduced this in 1.5. It is very easy to reproduce in devstack. Steps: 1. launch an instance: nova nova boot --image cirros-0.3.2-x86_64-uec --flavor 1 foo 2. call nova console-log nova console-log foo 3. restart rabbit sudo service rabbitmq-server restart 4. wait a couple of minutes for compute to reconnect 5. call nova console-log nova console-log foo It will fail the first time. Sometimes it fails the second time as well. Errors in the log: 2014-12-19 11:29:47.758 ERROR oslo.messaging._drivers.impl_rabbit [req-a69080c0-4d4c-4de4-9226-0cc5d89f98bc demo demo] Failed to publish message to topic 'compute.devstack': [Errno 104] Connection reset by peer 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 436, in _ensured 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit return fun(*args, **kwargs) 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 508, in __call__ 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit return fun(*args, channel=channels[0], **kwargs), channels[0] 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 714, in _publish 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit publisher = cls(self.conf, channel, topic=topic, **kwargs) 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 381, in __init__ 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit **options) 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 329, in __init__ 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit self.reconnect(channel) 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 337, in reconnect 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit routing_key=self.routing_key) 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 85, in __init__ 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit self.revive(self._channel) 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 218, in revive 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit self.declare() 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 105, in declare 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit self.exchange.declare() 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit nowait=nowait, passive=passive, 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 613, in exchange_declare 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit self._send_method((40, 10), args) 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 56, in _send_method 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit self.channel_id, method_sig, args, content, 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 221, in write_method 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit write_frame(1, channel, payload) 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/transport.py", line 182, in write_frame 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit frame_type, channel, size, payload, 0xce, 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio.py", line 359, in sendall 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit tail = self.send(data, flags) 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio.py", line 342, in send 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit total_sent += fd.send(data[total_sent:], flags) 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit error: [Errno 104] Connection reset by peer 2014-12-19 11:29:47.758 TRACE oslo.messaging._drivers.impl_rabbit 2014-12-19 11:29:47.760 ERROR oslo.messaging._drivers.impl_rabbit [req-a69080c0-4d4c-4de4-9226-0cc5d89f98bc demo demo] AMQP server on 192.168.50.131:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. 2014-12-19 11:29:48.773 ERROR oslo.messaging._drivers.impl_rabbit [req-a69080c0-4d4c-4de4-9226-0cc5d89f98bc demo demo] Failed to consume message from queue: [Errno 104] Connection reset by peer 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 436, in _ensured 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit return fun(*args, **kwargs) 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 508, in __call__ 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit return fun(*args, channel=channels[0], **kwargs), channels[0] 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 695, in _consume 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit return self.connection.drain_events(timeout=1) 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 275, in drain_events 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs) 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in drain_events 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit return connection.drain_events(**kwargs) 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 323, in drain_events 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(channel, args) 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 526, in _close 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit self._x_close_ok() 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 554, in _x_close_ok 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit self._send_method((10, 51)) 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 56, in _send_method 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit self.channel_id, method_sig, args, content, 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 221, in write_method 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit write_frame(1, channel, payload) 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/transport.py", line 182, in write_frame 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit frame_type, channel, size, payload, 0xce, 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio.py", line 359, in sendall 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit tail = self.send(data, flags) 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio.py", line 342, in send 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit total_sent += fd.send(data[total_sent:], flags) 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit error: [Errno 104] Connection reset by peer 2014-12-19 11:29:48.773 TRACE oslo.messaging._drivers.impl_rabbit 2014-12-19 11:29:48.774 ERROR oslo.messaging._drivers.impl_rabbit [req-a69080c0-4d4c-4de4-9226-0cc5d89f98bc demo demo] AMQP server on 192.168.50.131:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. 2014-12-19 11:30:48.826 ERROR nova.api.openstack [req-a69080c0-4d4c-4de4-9226-0cc5d89f98bc demo demo] Caught error: Timed out waiting for a reply to message ID 03fdd6852f0847f0a0f292d021d8d6bb 2014-12-19 11:30:48.826 TRACE nova.api.openstack Traceback (most recent call last): 2014-12-19 11:30:48.826 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/__init__.py", line 125, in __call__ 2014-12-19 11:30:48.826 TRACE nova.api.openstack return req.get_response(self.application) 2014-12-19 11:30:48.826 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send 2014-12-19 11:30:48.826 TRACE nova.api.openstack application, catch_exc_info=False) 2014-12-19 11:30:48.826 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application 2014-12-19 11:30:48.826 TRACE nova.api.openstack app_iter = application(self.environ, start_response) 2014-12-19 11:30:48.826 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ 2014-12-19 11:30:48.826 TRACE nova.api.openstack return resp(environ, start_response) 2014-12-19 11:30:48.826 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 828, in __call__ 2014-12-19 11:30:48.826 TRACE nova.api.openstack return self._call_app(env, start_response) 2014-12-19 11:30:48.826 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 758, in _call_app 2014-12-19 11:30:48.826 TRACE nova.api.openstack return self._app(env, _fake_start_response) 2014-12-19 11:30:48.826 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ 2014-12-19 11:30:48.826 TRACE nova.api.openstack return resp(environ, start_response) 2014-12-19 11:30:48.826 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ 2014-12-19 11:30:48.826 TRACE nova.api.openstack return resp(environ, start_response) 2014-12-19 11:30:48.826 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__ 2014-12-19 11:30:48.826 TRACE nova.api.openstack response = self.app(environ, start_response) 2014-12-19 11:30:48.826 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ 2014-12-19 11:30:48.826 TRACE nova.api.openstack return resp(environ, start_response) 2014-12-19 11:30:48.826 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__ 2014-12-19 11:30:48.826 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs) 2014-12-19 11:30:48.826 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func 2014-12-19 11:30:48.826 TRACE nova.api.openstack return self.func(req, *args, **kwargs) 2014-12-19 11:30:48.826 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 964, in __call__ 2014-12-19 11:30:48.826 TRACE nova.api.openstack content_type, body, accept) 2014-12-19 11:30:48.826 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 1029, in _process_stack 2014-12-19 11:30:48.826 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args) 2014-12-19 11:30:48.826 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 1119, in dispatch 2014-12-19 11:30:48.826 TRACE nova.api.openstack return method(req=request, **action_args) 2014-12-19 11:30:48.826 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/compute/contrib/console_output.py", line 65, in get_console_output 2014-12-19 11:30:48.826 TRACE nova.api.openstack length) 2014-12-19 11:30:48.826 TRACE nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 226, in wrapped 2014-12-19 11:30:48.826 TRACE nova.api.openstack return func(self, context, target, *args, **kwargs) 2014-12-19 11:30:48.826 TRACE nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 207, in wrapped 2014-12-19 11:30:48.826 TRACE nova.api.openstack return function(self, context, instance, *args, **kwargs) 2014-12-19 11:30:48.826 TRACE nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 2910, in get_console_output 2014-12-19 11:30:48.826 TRACE nova.api.openstack instance=instance, tail_length=tail_length) 2014-12-19 11:30:48.826 TRACE nova.api.openstack File "/opt/stack/nova/nova/compute/rpcapi.py", line 440, in get_console_output 2014-12-19 11:30:48.826 TRACE nova.api.openstack instance=instance, tail_length=tail_length) 2014-12-19 11:30:48.826 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call 2014-12-19 11:30:48.826 TRACE nova.api.openstack retry=self.retry) 2014-12-19 11:30:48.826 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send 2014-12-19 11:30:48.826 TRACE nova.api.openstack timeout=timeout, retry=retry) 2014-12-19 11:30:48.826 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 436, in send 2014-12-19 11:30:48.826 TRACE nova.api.openstack retry=retry) 2014-12-19 11:30:48.826 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 425, in _send 2014-12-19 11:30:48.826 TRACE nova.api.openstack result = self._waiter.wait(msg_id, timeout) 2014-12-19 11:30:48.826 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 315, in wait 2014-12-19 11:30:48.826 TRACE nova.api.openstack reply, ending = self._poll_connection(msg_id, timer) 2014-12-19 11:30:48.826 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 264, in _poll_connection 2014-12-19 11:30:48.826 TRACE nova.api.openstack % msg_id) 2014-12-19 11:30:48.826 TRACE nova.api.openstack MessagingTimeout: Timed out waiting for a reply to message ID 03fdd6852f0847f0a0f292d021d8d6bb 2014-12-19 11:30:48.826 TRACE nova.api.openstack