Heat:Heat engine failed to connect to rabbitmq after the rabbitmq cluster was broken and recovered back

Build : R3.02 35 Kilo

Logs saved at :http://10.204.216.50/Docs/bugs/<bug-id>

Had the below setup with 3 config and 3 openstack nodes.

One config node was down for long time.Once the node was recovered , the rabbitmq cluster was shown proper in all the 3 nodes.

But heat engine could not connect to rabbitmq still.Any heat command was failing with the below error

work around: restart heat api

Logs
=====
2016-05-11 11:38:21.734 10035 INFO oslo_messaging._drivers.impl_rabbit [req-1f8106fa-7c95-4ebe-913f-1eeb7686e312 4ea8ed4d23bd41c1b0e85367be140386 1b18515d8f874b0bbe826d1ba4f9300f] The exchange Exchange reply_71a121a0f8ea463ea3066576f83448a4(direct) to send to reply_71a121a0f8ea463ea3066576f83448a4 doesn't exist yet, retrying...
2016-05-11 11:38:22.494 10058 ERROR oslo_messaging._drivers.impl_rabbit [req-98f58335-9c58-4a41-afe6-aee97df5fadd 4ea8ed4d23bd41c1b0e85367be140386 2f6860cf4ff94216a4fc94a88627b3b7] Failed to publish message to topic 'reply_71a121a0f8ea463ea3066576f83448a4': Exchange.declare: (404) NOT_FOUND - no exchange 'reply_71a121a0f8ea463ea3066576f83448a4' in vhost '/'
2016-05-11 11:38:22.495 10058 DEBUG oslo_messaging._drivers.impl_rabbit [req-98f58335-9c58-4a41-afe6-aee97df5fadd 4ea8ed4d23bd41c1b0e85367be140386 2f6860cf4ff94216a4fc94a88627b3b7] Exception _error_callback /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py:1100
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit Traceback (most recent call last):
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 871, in ensure
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit ret, channel = autoretry_method()
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 436, in _ensured
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit return fun(*args, **kwargs)
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 508, in __call__
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit return fun(*args, channel=channels[0], **kwargs), channels[0]
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 857, in execute_method
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit method()
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1103, in _publish
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit publisher.send(self, msg, timeout)
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 465, in send
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit timeout)
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 394, in send
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit routing_key=self.routing_key)
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 85, in __init__
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit self.revive(self._channel)
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 218, in revive
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit self.declare()
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 105, in declare
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit self.exchange.declare()
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit nowait=nowait, passive=passive,
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 620, in exchange_declare
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit (40, 11), # Channel.exchange_declare_ok
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 69, in wait
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit return self.dispatch_method(method_sig, args, content)
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 87, in dispatch_method
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit return amqp_method(self, args)
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 241, in _close
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit reply_code, reply_text, (class_id, method_id), ChannelError,
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit NotFound: Exchange.declare: (404) NOT_FOUND - no exchange 'reply_71a121a0f8ea463ea3066576f83448a4' in vhost '/'
2016-05-11 11:38:22.495 10058 TRACE oslo_messaging._drivers.impl_rabbit
Traceback (most recent call last):

Setup
=====
host1 = 'root@10.204.217.139'
host2 = 'root@10.204.217.140'
host3 = 'root@10.204.217.147'
host4 = 'root@10.204.217.144'
host5 = 'root@10.204.217.147'
host6 = 'root@10.204.217.148'
host7 = 'root@10.204.217.149'
host8 = 'root@10.204.217.150'
host9 = 'root@10.204.217.210'
host10 = 'root@10.204.217.217'
host11 = 'root@10.204.217.218'
host12 = 'root@10.204.217.220'
host13 = 'root@10.204.217.247'
host14 = 'root@10.204.217.248'
host15 = 'root@10.204.217.249'
host16 = 'root@10.204.217.118'
host17 = 'root@10.204.217.119'
host18 = 'root@10.204.217.120'
host19 = 'root@10.204.217.121'
host20 = 'root@10.204.217.122'
host21 = 'root@10.204.217.123'
host22 = 'root@10.204.217.124'
host23 = 'root@10.204.217.131'

ext_routers = [('blr-mx2', '10.204.216.245')]
router_asn = 64512
public_vn_rtgt = 30001
#public_vn_subnet = "10.204.219.72/29"

host_build = 'vjoshi@10.204.216.56'

env.roledefs = {
    'all': [host1, host2, host3, host4, host5, host6,host7, host8, host9, host10, host11, host12, host13, host14, host15,host16,host17,host18,host19,host20,host21,host22,host23],
    'cfgm': [host1, host2, host3],
    'openstack': [host4, host5, host6],
    'webui': [host1, host2, host3],
    'control': [host1, host2, host3],
    'compute': [host7, host8, host9, host10, host11, host12, host13, host14, host15,host16,host17,host18,host19,host20,host21,host22,host23],
    'collector': [host1, host2, host3],
    'database': [host1, host2, host3],
    'build': [host_build],
}