Comment 0 for bug 1465840

Delete stack failed with next error:
http://paste.openstack.org/show/296169/, at the same time other services (like nova on command nova delete instance_uuid works fine)

haproxy status for heat:
      # pxname svname stot ereq econ eresp status chkfail chkdown downtime iid
      heat-api FRONTEND 0 0 OPEN 16
      heat-api node-76 0 0 0 UP 0 0 0 16
      heat-api node-77 0 0 0 UP 0 0 0 16
      heat-api node-79 0 0 0 UP 0 0 0 16
      heat-api BACKEND 0 0 0 UP 0 0 16
  heat-api-cfn FRONTEND 0 0 OPEN 17
  heat-api-cfn node-76 0 0 0 UP 0 0 0 17
  heat-api-cfn node-77 0 0 0 UP 0 0 0 17
  heat-api-cfn node-79 0 0 0 UP 0 0 0 17
  heat-api-cfn BACKEND 0 0 0 UP 0 0 17
heat-api-cloudwatch FRONTEND 0 0 OPEN 18
heat-api-cloudwatch node-76 0 0 0 UP 0 0 0 18
heat-api-cloudwatch node-77 0 0 0 UP 0 0 0 18
heat-api-cloudwatch node-79 0 0 0 UP 0 0 0 18
heat-api-cloudwatch BACKEND 0 0 0 UP 0 0 18

crm status

http://paste.openstack.org/show/296170/

Steps to Reproduce:
Os: Centos
HA wit neutro gre:
1 controller + 2 controller with mongo
1 mongo
1 cinder
2 omputes
Ceilometer enabled

1.Deploy cluster, when cluster is ready
2. Navigate to fuel health tab and run all ostf tests - ha, smoke. sanity, platfrom tests are passed (configuration may fail if you do not change default cread for ssh on master node and user cred to openstack cluster)
3. Shutdown primary controller
4. wait 5 minutes - run ostf ha suit (it passed, if not you may need to wait for a liitle bit more and run again)
5. run smoke, sanity OSTF tests - they are passed
6. Run platfrom tests - Actual result all heat test failed (update/ create/ delete stack)
with 504 error
7. turn on controller - repeat step 4-6 - result is the same all heat tests are failed

ssh to each controller and try to delete stack - failed with error listed about

{"build_id": "2015-06-08_06-13-27", "build_number": "521", "release_versions": {"2014.2.2-6.1": {"VERSION": {"build_id": "2015-06-08_06-13-27", "build_number": "521", "api": "1.0", "fuel-library_sha": "f43c2ae1af3b493ee0e7810eab7bb7b50c986c7d", "nailgun_sha": "4340d55c19029394cd5610b0e0f56d6cb8cb661b", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "4fc55db0265bbf39c369df398b9dc7d6469ba13b", "astute_sha": "7766818f079881e2dbeedb34e1f67e517ed7d479", "fuel-ostf_sha": "7c938648a246e0311d05e2372ff43ef1eb2e2761", "release": "6.1", "fuelmain_sha": "bcc909ffc5dd5156ba54cae348b6a07c1b607b24"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "f43c2ae1af3b493ee0e7810eab7bb7b50c986c7d", "nailgun_sha": "4340d55c19029394cd5610b0e0f56d6cb8cb661b", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "4fc55db0265bbf39c369df398b9dc7d6469ba13b", "astute_sha": "7766818f079881e2dbeedb34e1f67e517ed7d479", "fuel-ostf_sha": "7c938648a246e0311d05e2372ff43ef1eb2e2761", "release": "6.1", "fuelmain_sha": "bcc909ffc5dd5156ba54cae348b6a07c1b607b24"}

Second env:
UBUNTU
3 controllers with ceph + 1 compute + 2 mongo + 2 ceph
with nova Flat
1. Deploy cluster
2. When cluster ready - run ostf ha, smoke, sanity, platfrom tests - tests are passed
3. shutdown non-primary controller
4. wait near 5-7 minutes and run ostf ha - passed
5. run sanity/ smoke - passed
6. run platform tests - heat tests are failed with 504 error
7. ssh to online controllers and try to create and delete stack - failed with 504 error

Info: Add snapshot later according it is to big and to slow uploading in google drive