Heat provisioning on liberty is failing on HA setup

contrail version : 3.0.2.0-32

Having 3 nodes running (openstack + config + control ) and 1 compute node.

2016-04-22 14:35:07:845409: [root@10.84.24.32] out: + openstack-config --del /etc/heat/heat.conf database connection
2016-04-22 14:35:07:845427: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf DEFAULT sql_connection mysql://heat:c0ntrail123@127.0.0.1/heat
2016-04-22 14:35:07:877653: [root@10.84.24.32] out: + '[' 10.84.24.201 '!=' none ']'
2016-04-22 14:35:07:877719: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf DEFAULT sql_connection mysql://heat:c0ntrail123@10.84.24.201:33306/heat
2016-04-22 14:35:07:877742: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf database idle_timeout 180
2016-04-22 14:35:07:893164: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf database min_pool_size 100
2016-04-22 14:35:07:924722: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf database max_pool_size 350
2016-04-22 14:35:07:924801: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf database max_overflow 700
2016-04-22 14:35:07:940276: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf database retry_interval 5
2016-04-22 14:35:07:955729: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf database max_retries -1
2016-04-22 14:35:07:971164: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf database db_max_retries 3
2016-04-22 14:35:07:986631: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf database db_retry_interval 1
2016-04-22 14:35:07:994005: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf database connection_debug 10
2016-04-22 14:35:08:009449: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf database pool_timeout 120
2016-04-22 14:35:08:024919: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf DEFAULT rabbit_retry_interval 10
2016-04-22 14:35:08:040380: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf DEFAULT rabbit_retry_backoff 5
2016-04-22 14:35:08:055864: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf DEFAULT rabbit_max_retries 0
2016-04-22 14:35:08:071331: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf DEFAULT rabbit_ha_queues True
2016-04-22 14:35:08:078692: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf heat_api bind_port 8005
2016-04-22 14:35:08:094134: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf DEFAULT rpc_backend heat.openstack.common.rpc.impl_kombu
2016-04-22 14:35:08:109614: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf DEFAULT rabbit_host 10.84.24.201
2016-04-22 14:35:08:125077: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf DEFAULT rabbit_port 5673
2016-04-22 14:35:08:140540: [root@10.84.24.32] out: ++ python -c 'from distutils.sysconfig import get_python_lib; print(get_python_lib())'
2016-04-22 14:35:08:156060: [root@10.84.24.32] out: + PYDIST=/usr/lib/python2.7/dist-packages
2016-04-22 14:35:08:187799: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf DEFAULT plugin_dirs /usr/lib/python2.7/dist-packages/vnc_api/gen/heat/resources
2016-04-22 14:35:08:187941: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf DEFAULT heat_waitcondition_server_url http://10.84.24.201:8000/v1/waitcondition
2016-04-22 14:35:08:203521: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf keystone_authtoken auth_uri http://10.84.24.201:5000/v2.0
2016-04-22 14:35:08:218932: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf keystone_authtoken auth_host 10.84.24.201
2016-04-22 14:35:08:234399: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf keystone_authtoken auth_port 35357
2016-04-22 14:35:08:249769: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf keystone_authtoken auth_protocol http
2016-04-22 14:35:08:265201: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf keystone_authtoken admin_tenant_name service
2016-04-22 14:35:08:280660: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf keystone_authtoken admin_user heat
2016-04-22 14:35:08:296184: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf keystone_authtoken admin_password contrail123
2016-04-22 14:35:08:311652: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf clients_contrail user admin
2016-04-22 14:35:08:319038: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf clients_contrail password contrail123
2016-04-22 14:35:08:334479: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf clients_contrail tenant admin
2016-04-22 14:35:08:349993: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf clients_contrail api_server 10.84.24.32
2016-04-22 14:35:08:365420: [root@10.84.24.32] out: + '[' 10.84.24.201 '!=' none ']'
2016-04-22 14:35:08:380858: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf clients_contrail api_server 10.84.24.201
2016-04-22 14:35:08:380914: [root@10.84.24.32] out: + openstack-config --set /etc/heat/heat.conf clients_contrail api_base_url /
2016-04-22 14:35:08:396360: [root@10.84.24.32] out: + for APP in heat
2016-04-22 14:35:08:411782: [root@10.84.24.32] out: + '[' 1 -eq 1 ']'
2016-04-22 14:35:08:411857: [root@10.84.24.32] out: + openstack-db -y --init --service heat --password c0ntrail123 --rootpw 1432b6ac99a56b483362
2016-04-22 14:35:08:411929: [root@10.84.24.32] out: + systemctl --version
2016-04-22 14:35:08:411997: [root@10.84.24.32] out: + '[' -f /etc/redhat-release ']'
2016-04-22 14:35:08:412054: [root@10.84.24.32] out: + '[' -f /etc/lsb-release ']'
2016-04-22 14:35:08:412119: [root@10.84.24.32] out: + egrep -q 'DISTRIB_ID.*Ubuntu' /etc/lsb-release
2016-04-22 14:35:08:412177: [root@10.84.24.32] out: + is_ubuntu=1
2016-04-22 14:35:08:412225: [root@10.84.24.32] out: + is_redhat=0
2016-04-22 14:35:08:412255: [root@10.84.24.32] out: + web_svc=apache2
2016-04-22 14:35:08:412275: [root@10.84.24.32] out: + mysql_svc=mysql
2016-04-22 14:35:08:412295: [root@10.84.24.32] out: + mysql_pkg_chk='expr 1'
2016-04-22 14:35:08:412315: [root@10.84.24.32] out: + dbmanage_cmd='"$APP-manage $db_sync"'
2016-04-22 14:35:08:412334: [root@10.84.24.32] out: + '[' 8 -gt 0 ']'
2016-04-22 14:35:08:412354: [root@10.84.24.32] out: + case "$1" in
2016-04-22 14:35:08:412373: [root@10.84.24.32] out: + ASSUME_YES=yes
2016-04-22 14:35:08:412393: [root@10.84.24.32] out: + shift
2016-04-22 14:35:08:412422: [root@10.84.24.32] out: + '[' 7 -gt 0 ']'
2016-04-22 14:35:08:412440: [root@10.84.24.32] out: + case "$1" in
016-04-22 14:35:08:412488: [root@10.84.24.32] out: + shift
2016-04-22 14:35:08:412508: [root@10.84.24.32] out: + '[' 6 -gt 0 ']'
2016-04-22 14:35:08:412527: [root@10.84.24.32] out: + case "$1" in
2016-04-22 14:35:08:412546: [root@10.84.24.32] out: + shift
2016-04-22 14:35:08:412565: [root@10.84.24.32] out: + APP=heat
2016-04-22 14:35:08:412585: [root@10.84.24.32] out: + shift
2016-04-22 14:35:08:412605: [root@10.84.24.32] out: + '[' 4 -gt 0 ']'
2016-04-22 14:35:08:412624: [root@10.84.24.32] out: + case "$1" in
2016-04-22 14:35:08:412643: [root@10.84.24.32] out: + shift
2016-04-22 14:35:08:412662: [root@10.84.24.32] out: + MYSQL_APP_PW=c0ntrail123
2016-04-22 14:35:08:412682: [root@10.84.24.32] out: + shift
2016-04-22 14:35:08:412701: [root@10.84.24.32] out: + '[' 2 -gt 0 ']'
2016-04-22 14:35:08:412721: [root@10.84.24.32] out: + case "$1" in
2016-04-22 14:35:08:412740: [root@10.84.24.32] out: + shift
2016-04-22 14:35:08:412760: [root@10.84.24.32] out: + MYSQL_ROOT_PW=1432b6ac99a56b483362
2016-04-22 14:35:08:412787: [root@10.84.24.32] out: + shift
2016-04-22 14:35:08:412808: [root@10.84.24.32] out: + '[' 0 -gt 0 ']'
2016-04-22 14:35:08:412836: [root@10.84.24.32] out: + '[' '!' init ']'
2016-04-22 14:35:08:412856: [root@10.84.24.32] out: + '[' '!' heat ']'
2016-04-22 14:35:08:412876: [root@10.84.24.32] out: + case "$APP" in
2016-04-22 14:35:08:412894: [root@10.84.24.32] out: + MYSQL_APP_PW_DEFAULT=heat
2016-04-22 14:35:08:412924: [root@10.84.24.32] out: + : c0ntrail123
2016-04-22 14:35:08:412959: [root@10.84.24.32] out: + '[' heat = glance ']'
2016-04-22 14:35:08:412997: [root@10.84.24.32] out: + APP_CONFIG=/etc/heat/heat.conf
2016-04-22 14:35:08:413032: [root@10.84.24.32] out: + NEW_MYSQL_INSTALL=0
2016-04-22 14:35:08:413066: [root@10.84.24.32] out: + expr 1
2016-04-22 14:35:08:413099: [root@10.84.24.32] out: 1
2016-04-22 14:35:08:413133: [root@10.84.24.32] out: + service_running mysql
2016-04-22 14:35:08:413166: [root@10.84.24.32] out: + test ''
2016-04-22 14:35:08:413214: [root@10.84.24.32] out: + service mysql status
2016-04-22 14:35:08:413253: [root@10.84.24.32] out: + '[' 0 -eq 1 ']'
2016-04-22 14:35:08:445891: [root@10.84.24.32] out: + '[' '!' defined ']'
2016-04-22 14:35:08:445956: [root@10.84.24.32] out: + MYSQL_ROOT_PW_ARG=
2016-04-22 14:35:08:445983: [root@10.84.24.32] out: + '[' defined ']'
2016-04-22 14:35:08:446006: [root@10.84.24.32] out: + MYSQL_ROOT_PW_ARG=--password=1432b6ac99a56b483362
2016-04-22 14:35:08:446026: [root@10.84.24.32] out: + echo 'SELECT 1;'
2016-04-22 14:35:08:446048: [root@10.84.24.32] out: + mysql -u root --password=1432b6ac99a56b483362
2016-04-22 14:35:08:446069: [root@10.84.24.32] out: + echo 'Verified connectivity to MySQL.'
2016-04-22 14:35:08:446090: [root@10.84.24.32] out: Verified connectivity to MySQL.
2016-04-22 14:35:08:446111: [root@10.84.24.32] out: + '[' init = init ']'
2016-04-22 14:35:08:446132: [root@10.84.24.32] out: ++ echo 'SELECT COUNT(*) FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME='\''heat'\'';'
2016-04-22 14:35:08:446154: [root@10.84.24.32] out: ++ mysql -u root --password=1432b6ac99a56b483362
2016-04-22 14:35:08:446175: [root@10.84.24.32] out: ++ tail -n+2
2016-04-22 14:35:08:446196: [root@10.84.24.32] out: + dbs=0
2016-04-22 14:35:08:446216: [root@10.84.24.32] out: + '[' 0 '!=' 0 ']'
2016-04-22 14:35:08:446242: [root@10.84.24.32] out: ++ echo 'SELECT COUNT(*) FROM mysql.user WHERE User = '\''heat'\'';'
2016-04-22 14:35:08:446264: [root@10.84.24.32] out: ++ mysql -u root --password=1432b6ac99a56b483362
2016-04-22 14:35:08:446285: [root@10.84.24.32] out: ++ tail -n+2
2016-04-22 14:35:08:446305: [root@10.84.24.32] out: + users=0
2016-04-22 14:35:08:446326: [root@10.84.24.32] out: + '[' 0 '!=' 0 ']'
2016-04-22 14:35:08:446347: [root@10.84.24.32] out: + '[' init = init ']'
2016-04-22 14:35:08:446468: [root@10.84.24.32] out: + '[' c0ntrail123 '!=' heat ']'
2016-04-22 14:35:08:446488: [root@10.84.24.32] out: + echo 'Updating '\''heat'\'' database password in /etc/heat/heat.conf'
2016-04-22 14:35:08:446509: [root@10.84.24.32] out: Updating 'heat' database password in /etc/heat/heat.conf
2016-04-22 14:35:08:446530: [root@10.84.24.32] out: + sed -i -e 's|^#* *\(\(sql_\)*connection *= *mysql://heat\):.*@|\1:c0ntrail123@|' /etc/heat/heat.conf
2016-04-22 14:35:08:446551: [root@10.84.24.32] out: + echo 'Initializing the heat database, please wait...'
2016-04-22 14:35:08:446572: [root@10.84.24.32] out: Initializing the heat database, please wait...
2016-04-22 14:35:08:446592: [root@10.84.24.32] out: + '[' heat = nova ']'
2016-04-22 14:35:08:446612: [root@10.84.24.32] out: + '[' heat = cinder ']'
2016-04-22 14:35:08:446632: [root@10.84.24.32] out: + db_sync=db_sync
2016-04-22 14:35:08:446652: [root@10.84.24.32] out: + '[' 0 -eq 1 ']'
2016-04-22 14:35:08:446671: [root@10.84.24.32] out: + '[' 1 -eq 1 ']'
2016-04-22 14:35:08:446691: [root@10.84.24.32] out: + heat-manage db_sync
2016-04-22 14:35:08:446711: [root@10.84.24.32] out: 2016-04-22 14:35:08.813 16160 WARNING oslo_config.cfg [-] Option "sql_connection" from group "DEFAULT" is deprecated. Use option "connection" from group "database".
2016-04-22 14:35:08:847882: [root@10.84.24.32] out: 2016-04-22 14:35:09.018 16160 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -1 attempts left.
2016-04-22 14:35:09:048504: [root@10.84.24.32] out: 2016-04-22 14:35:15.015 16160 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -2 attempts left.
2016-04-22 14:35:15:025871: [root@10.84.24.32] out: 2016-04-22 14:35:23.014 16160 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -3 attempts left.
2016-04-22 14:35:23:045799: [root@10.84.24.32] out: 2016-04-22 14:35:31.015 16160 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -4 attempts left.
2016-04-22 14:35:31:029586: [root@10.84.24.32] out: 2016-04-22 14:35:39.014 16160 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -5 attempts left.
2016-04-22 14:35:39:048323: [root@10.84.24.32] out: 2016-04-22 14:35:47.017 16160 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -6 attempts left.
2016-04-22 14:35:47:030434: [root@10.84.24.32] out: 2016-04-22 14:35:55.015 16160 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -7 attempts left.
2016-04-22 14:35:55:050804: [root@10.84.24.32] out: 2016-04-22 14:36:01.018 16160 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -8 attempts left.
2016-04-22 14:36:01:028945: [root@10.84.24.32] out: 2016-04-22 14:36:11.024 16160 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -9 attempts left.
2016-04-22 14:36:11:049466: [root@10.84.24.32] out: 2016-04-22 14:36:21.026 16160 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -10 attempts left.
2016-04-22 14:36:21:036609: [root@10.84.24.32] out: 2016-04-22 14:36:34.038 16160 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -11 attempts left.
2016-04-22 14:36:34:065461: [root@10.84.24.32] out: 2016-04-22 14:36:42.042 16160 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -12 attempts left.
2016-04-22 14:36:42:048617: [root@10.84.24.32] out: 2016-04-22 14:36:50.039 16160 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -13 attempts left.
2016-04-22 14:36:50:067287: [root@10.84.24.32] out: 2016-04-22 14:36:56.039 16160 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -14 attempts left.
2016-04-22 14:36:56:044880: [root@10.84.24.32] out: 2016-04-22 14:37:04.038 16160 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -15 attempts left.
2016-04-22 14:37:04:060307: [root@10.84.24.32] out: 2016-04-22 14:37:14.039 16160 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -16 attempts left.
2016-04-22 14:37:14:045680: [root@10.84.24.32] out: 2016-04-22 14:37:22.042 16160 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -17 attempts left.
2016-04-22 14:37:22:063750: [root@10.84.24.32] out: 2016-04-22 14:37:30.046 16160 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -18 attempts left.
2016-04-22 14:37:30:083394: [root@10.84.24.32] out: 2016-04-22 14:37:38.049 16160 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -19 attempts left.
2016-04-22 14:37:38:065803: [root@10.84.24.32] out: 2016-04-22 14:37:46.050 16160 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -20 attempts left.
2016-04-22 14:37:46:084896: [root@10.84.24.32] out: 2016-04-22 14:37:54.050 16160 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -21 attempts left.
2016-04-22 14:37:54:068293: [root@10.84.24.32] out: 2016-04-22 14:38:02.052 16160 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -22 attempts left.
2016-04-22 14:38:02:086703: [root@10.84.24.32] out: 2016-04-22 14:38:10.051 16160 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -23 attempts left.
2016-04-22 14:38:10:065401: [root@10.84.24.32] out: 2016-04-22 14:38:18.050 16160 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -24 attempts left.
2016-04-22 14:38:18:085643: [root@10.84.24.32] out: 2016-04-22 14:38:26.056 16160 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -25 attempts left.
2016-04-22 14:38:26:068904: [root@10.84.24.32] out: 2016-04-22 14:38:34.055 16160 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -26 attempts left.
2016-04-22 14:38:34:087632: [root@10.84.24.32] out: 2016-04-22 14:38:40.051