Heat operation failed after controller failover

https://drive.google.com/a/mirantis.com/file/d/0B_tSitrwrgvoYjh1ek95S08zTHc/view?usp=sharing

Attached snapshot does not contain heat related logs. Will sync with Tanya to get more info.

yep you are right, for some reason remote directory for nodes are empty :( upload snapshot from second env

https://drive.google.com/a/mirantis.com/file/d/0B_tSitrwrgvoRDlwY0FWdDY3SDA/view?usp=sharing

Regarding last snapshot all heat errors are related with issues with rabbit mq (heat-api could not connect to rabbit and find queue).

Also ostf logs have errors of other (not heat service) tests [1]

Regarding issues mentioned above is not really possible to find root cause of errors and it looks like other services (not only Heat) are broken too.

After discussion with tatyana-leontovich, we decided to deploy one separate environment for reproducing this issue. It should give ability to get clear logs which show root problem.

Note: Deployment will be ready tomorrow.

[1] http://paste.openstack.org/show/298982/

We investigated the issue and looks like the root of the issue in the following: we have disabled ha_queues in heat.conf and after the disaster scenario Heat lost one queue and can't work until restart. We need to try to reproduce the issue with the fix for the issue and if it helps we need to change this parameter in Heat configuration file.

We need to change "rabbit_ha_queues=False " to "rabbit_ha_queues=True" in heat config file to fix the issue.
Then we need to restart Heat services on all controllers:

root@node-1:~# service heat-api restart
root@node-1:~# service heat-engine restart
root@node-1:~# service heat-api-cfn restart
root@node-1:~# service heat-api-cloudwatch restart

Tried to reproduce issue with

1. Find "primary controller":
root@node-2:~# ifconfig | grep br-ex-hapr
br-ex-hapr Link encap:Ethernet HWaddr 06:7a:c8:57:99:3b
2. Shutdown primary controller:
shutdown now -P
3. Wait 10 minutes
4. Run Heat OSTF tests

Observed Result:
Issue not reproduced with parameter "rabbit_ha_queues=True" in /etc/heat/heat.conf, so, it is a way to fix the issue, I will commit fix soon.

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/193488

Bug in upstream puppet modules:
https://bugs.launchpad.net/puppet-heat/+bug/1466810

High bugs cannot be addressed while HCF in progress

This bug impacts HA failover of Heat services badly, rasing to critical.

It seems that this issue affects only 1 controller node cases, which is not HA and it is not important which value for the rabbit_ha_queues will be configured for services.

I checked the 1 node deploy:
/etc/heat/heat.conf:rabbit_ha_queues=False
/etc/keystone/keystone.conf:rabbit_ha_queues=True
/etc/hiera/globals.yaml: rabbit_ha_queues: true
/etc/nova/nova.conf:rabbit_ha_queues=True
/etc/neutron/neutron.conf:rabbit_ha_queues=True
/etc/cinder/cinder.conf:rabbit_ha_queues=True
/etc/glance/glance-api.conf:rabbit_ha_queues=True

yes, for heat is seems inconsistent, but can be ignored for 1 controller node case, AFAIK.

I'm testing the case with 2 controllers, and will provide results for the heat ha queueus configuration

Yes, the bug looks valid.
For rabbit_hosts=10.109.2.5:5673, 10.109.2.6:5673
heat.conf still contains rabbit_ha_queues=False
which is a critical issue

This bug is not in heat module. This is about how we pass amqp hosts to the heat class

Related bug https://bugs.launchpad.net/fuel/+bug/1466833

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/193513

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/193513
Reason: master first

Change abandoned by Timur Nurlygayanov (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/193488
Reason: We found the root of this issue, we fixed it with another commit: https://review.openstack.org/#/c/193511/

Change abandoned by Timur Nurlygayanov (<email address hidden>) on branch: master
Review: https://review.openstack.org/193483
Reason: We found the root of this issue, we fixed it with another commit: https://review.openstack.org/#/c/193511/

Release Notes:
Default Heat configuration for HA environments is not correct, we need to change parameter "rabbit_ha_queues=True" in /etc/heat/heat.conf to avoid Heat failures in case of destructive scenarios ( when we will power off some controllers and etc.)
When Heat configuration files will be changed on all controllers we should manually restart all Heat services on all OpenStack controllers:

:~# service heat-api restart
:~# service heat-engine restart
:~# service heat-api-cfn restart
:~# service heat-api-cloudwatch restart

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/193513

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/199447

Here is the correct fix for MOS 6.1:
https://review.openstack.org/199447

Can't reproduce on 7.0-26 with both scenarios.

{"build_id": "2015-07-06_18-08-24", "build_number": "26", "release_versions": {"2014.2.2-7.0": {"VERSION": {"build_id": "2015-07-06_18-08-24", "build_number": "26", "api": "1.0", "fuel-library_sha": "251c54e8de2f41aacd260751e7a891e9fbffc45d", "nailgun_sha": "d040c5cebc9cdd24ef20cb7ecf0a337039baddec", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-7.0", "production": "docker", "python-fuelclient_sha": "315d8bf991fbe7e2ab91abfc1f59b2f24fd92f45", "astute_sha": "9cbb8ae5adbe6e758b24b3c1021aac1b662344e8", "fuel-ostf_sha": "a752c857deafd2629baf646b1b3188f02ff38084", "release": "7.0", "fuelmain_sha": "4f2dff3bdc327858fa45bcc2853cfbceae68a40c"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "251c54e8de2f41aacd260751e7a891e9fbffc45d", "nailgun_sha": "d040c5cebc9cdd24ef20cb7ecf0a337039baddec", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-7.0", "production": "docker", "python-fuelclient_sha": "315d8bf991fbe7e2ab91abfc1f59b2f24fd92f45", "astute_sha": "9cbb8ae5adbe6e758b24b3c1021aac1b662344e8", "fuel-ostf_sha": "a752c857deafd2629baf646b1b3188f02ff38084", "release": "7.0", "fuelmain_sha": "4f2dff3bdc327858fa45bcc2853cfbceae68a40c"}

verified
{"build_id": "286", "build_number": "286", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "286", "build_number": "286", "api": "1.0", "fuel-library_sha": "ff63a0bbc93a3a0fb78215c2fd0c77add8dfe589", "nailgun_sha": "5c33995a2e6d9b1b8cdddfa2630689da5084506f", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d", "production": "docker", "python-fuelclient_sha": "1ce8ecd8beb640f2f62f73435f4e18d1469979ac", "astute_sha": "8283dc2932c24caab852ae9de15f94605cc350c6", "fuel-ostf_sha": "1f08e6e71021179b9881a824d9c999957fcc7045", "release": "7.0", "fuelmain_sha": "9ab01caf960013dc882825dc9b0e11ccf0b81cb0"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "ff63a0bbc93a3a0fb78215c2fd0c77add8dfe589", "nailgun_sha": "5c33995a2e6d9b1b8cdddfa2630689da5084506f", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d", "production": "docker", "python-fuelclient_sha": "1ce8ecd8beb640f2f62f73435f4e18d1469979ac", "astute_sha": "8283dc2932c24caab852ae9de15f94605cc350c6", "fuel-ostf_sha": "1f08e6e71021179b9881a824d9c999957fcc7045", "release": "7.0", "fuelmain_sha": "9ab01caf960013dc882825dc9b0e11ccf0b81cb0"}

Related fix proposed to branch: master
Review: https://review.openstack.org/223455

Reviewed: https://review.openstack.org/223455
Committed: https://git.openstack.org/cgit/stackforge/fuel-docs/commit/?id=ed673f381afea23ac9becf0adfb770ea7129cb4a
Submitter: Jenkins
Branch: master

commit ed673f381afea23ac9becf0adfb770ea7129cb4a
Author: evkonstantinov <email address hidden>
Date: Tue Sep 15 10:25:16 2015 +0300

    Add Heat HA false issue to relnotes

    Change-Id: Ifbacce627862b25142e2bae945ee8c434900360b
    Related-Bug:#1465840

Change abandoned by Sergii Golovatiuk (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/199447