Nova crashes with DatabaseNotControlledError

It was something wrong with WSREP during deployment:

http://paste.openstack.org/show/591124/

That cause incomplete migration of nova db

That is the real scenario and real cause of that problem:

Facts:

1. We have primary-openstack-controller task.
2. That job is repeatable. If it fail it will be re-triggered.
3. That job triggers Nova installation and configuration
4. Installation and configuration triggers sync-db.
5. sync-db can't repeat step if something goes wrong with connection to DB.

So that is the fail scenario:

1. sync-db was working while wsrep was starting and re-balancing nodes.
2. mysql node on primary controller had failed while sync-db was running
3. sync-db was not able to recover and failed
4. primary-openstack-controller task was failed too.
5. primary-openstack-controller task was triggered one more time.
6. nova installation was not triggered because nova is already installed and configured.
7. sync-db was not triggered too because it depends on nova installation process.
8. primary-openstack-controller task was succeed because there where not fails.
9. so deployment has success state too
10. but nova database had not filled with data and was broken.

Possible solutions:
1. Allways trigger sync-db on primary-openstack-controller.
2. Add check about current db state.
3. Somehow fix wrep and mysql starting procedure.
4. Fix sync-db to repeat steps.

That problem already fixed in master https://review.openstack.org/#/c/379217/

Fix created for stable/mitaka https://review.openstack.org/#/c/406104

Fix for stable/newton https://review.openstack.org/#/c/406138

Fix that was created for bug #1628580 is incomplete.

https://review.openstack.org/#/c/406104 has been merged
https://review.openstack.org/#/c/406138 has been merged

There are a number of fixes that has been merged to Mitaka and Newton for various components:

Aodh:
https://review.openstack.org/407069
https://review.openstack.org/407070

Ceilometer:
https://review.openstack.org/407074
https://review.openstack.org/407073

Cinder:
https://review.openstack.org/407077
https://review.openstack.org/407076

Glance:
https://review.openstack.org/407078
https://review.openstack.org/407080

Heat:
https://review.openstack.org/407081
https://review.openstack.org/407082

Ironic:
https://review.openstack.org/407083
https://review.openstack.org/407084

Neutron:
https://review.openstack.org/407090
https://review.openstack.org/407091

Murano:
https://review.openstack.org/407087
https://review.openstack.org/407088

Sahara:
https://review.openstack.org/407093
https://review.openstack.org/407092

Keystone for Newton:
https://review.openstack.org/407086

Keystone for Mitaka still in progress.

While original issue has been fixed we intend to care about Keystone patch to Mitaka in separate bug:
https://bugs.launchpad.net/fuel/+bug/1655968

Verified on 9.2 snapshot #771.

Latest runs with fix are passed since #724 snapshot (#171-#179 runs):
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.plugins.fuel_plugin_example/

This issue was fixed in the openstack/puppet-aodh 9.5.0 release.

This issue was fixed in the openstack/puppet-ceilometer 9.5.0 release.

This issue was fixed in the openstack/puppet-cinder 9.5.0 release.

This issue was fixed in the openstack/puppet-glance 9.5.0 release.

This issue was fixed in the openstack/puppet-heat 9.5.0 release.

This issue was fixed in the openstack/puppet-ironic 9.5.0 release.

This issue was fixed in the openstack/puppet-keystone 9.5.0 release.

This issue was fixed in the openstack/puppet-murano 9.5.0 release.

This issue was fixed in the openstack/puppet-neutron 9.5.0 release.

This issue was fixed in the openstack/puppet-nova 9.5.0 release.

This issue was fixed in the openstack/puppet-sahara 9.5.0 release.

This issue was fixed in the openstack/puppet-barbican 9.6.0 release.