N ->O upgrade: after running major-upgrade-composable-steps.yaml nova-api cannot connect to Galera on 2/3 controllers

So after a discussion with Damien and Michele, the problem was found.

It appears that when the nova cell is create it hardcode into the database the connection parameter present in /etc/nova/nova.conf of the node where it's run. Running it on controller0 for instance will give you this in the database:

    mysql+pymysql://nova:c2cdagE8PyAbnpers3AD88Hge@10.0.0.19nova?bind_address=10.0.0.20'

This is later used to create a connection to the database for nova cell information. This obviously fails on 2 other node as they don't have the 10.0.0.20 address.

To prevent this issue, this workaround have been done: https://review.openstack.org/#/c/436192/ removing the bind_address parameter from the configuration line.

The sequence of event on the seems correct:
 1. update hiera data;
 2. create nova cell with database option;

From the journalctl logs:

    Mar 22 19:23:44 overcloud-controller-0.localdomain os-collect-config[4197]: [2017-03-22 19:23:44,806] (heat-config) [DEBUG] Running /usr/libexec/heat-config/hooks/hiera < /var/lib/heat-config/deployed/e4e9bd8e-4b7b-41da-b040-d4f563f2fd48.json
    Mar 22 19:23:44 overcloud-controller-0.localdomain os-collect-config[4197]: [2017-03-22 19:23:44,852] (heat-config) [DEBUG] Running heat-config-notify /var/lib/heat-config/deployed/e4e9bd8e-4b7b-41da-b040-d4f563f2fd48.json < /var/lib/heat-config/deployed/e4e9bd8e-4b7b-41da-b040-d4f563f2fd48.notify.json

    $ grep nova::database_connection /var/lib/heat-config/deployed/e4e9bd8e-4b7b-41da-b040-d4f563f2fd48.json
            "nova::database_connection": "mysql+pymysql://nova:c2cdagE8PyAbnpers3AD88Hge@10.0.0.19/nova?read_default_file=/etc/my.cnf.d/tripleo.cnf&read_default_group=tripleo",

    [root@overcloud-controller-0 e]# journalctl | grep 'nova-manage cell_v2'
    Mar 22 19:39:52 overcloud-controller-0.localdomain ansible-command[440226]: Invoked with warn=True executable=None _uses_shell=False _raw_params=nova-manage cell_v2 map_cell0 removes=None creates=None chdir=None
    Mar 22 19:39:55 overcloud-controller-0.localdomain ansible-command[440632]: Invoked with warn=True executable=None _uses_shell=True _raw_params=nova-manage cell_v2 create_cell --name='default' --database_connection=$(hiera nova::database_connection) removes=None creates=None chdir=None
    Mar 22 19:40:09 overcloud-controller-0.localdomain ansible-command[443480]: Invoked with warn=True executable=None _uses_shell=False _raw_params=nova-manage cell_v2 map_cell_and_hosts removes=None creates=None chdir=None
    Mar 22 19:40:12 overcloud-controller-0.localdomain ansible-command[443950]: Invoked with warn=True executable=None _uses_shell=True _raw_params=nova-manage cell_v2 list_cells | sed -e '1,3d' -e '$d' | awk -F ' *| *' '$2 == "default" {print $4}' removes=None creates=None chdir=None
    Mar 22 19:40:15 overcloud-controller-0.localdomain ansible-command[444382]: Invoked with warn=True executable=None _uses_shell=False _raw_params=nova-manage cell_v2 map_instances --cell_uuid 7f04f00d-4b9d-478d-941f-d93a7be145e7 removes=None creates=None chdir=None

    INSERT INTO `cell_mappings` VALUES
    ('2017-03-22 19:39:54',NULL,2,'00000000-0000-0000-0000-000000000000','cell0','none:/...

Fix proposed to branch: master
Review: https://review.openstack.org/449080

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/449093

For the NULL entry another launchpad bug has been created : https://bugs.launchpad.net/tripleo/+bug/1675418

Reviewed: https://review.openstack.org/449080
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=4883e8b229c7fa3b8fe828bd3e06aa16e852d95c
Submitter: Jenkins
Branch: master

commit 4883e8b229c7fa3b8fe828bd3e06aa16e852d95c
Author: Sofer Athlan-Guyot <email address hidden>
Date: Thu Mar 23 12:10:48 2017 +0100

    [N->O] Fix wrong database connection for cell0 during upgrade.

    During upgrade the cell0 database has the connection pointing to

       mysql+pymysql://nova:c2cdagE8PyAbnpers3AD88Hge@10.0.0.19/nova_cell0?bind_address=10.0.0.20

    where 10.0.0.20 was the ip of the bootstrap node. This makes the
    nova-api fails on 2/3 node at the end of the
    major-upgrade-composable-steps.yaml step.

    We do have the right value in the hiera database so make sure we use
    it for cell0 creation and not the nova.conf file which hasn't been
    updated yet.

    Change-Id: I09775206cb8fc5e15934f7e4475506a7fe17271e
    Closes-Bug: #1675359

Reviewed: https://review.openstack.org/449093
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=d99a06705489b1d63d72b0f2296d7236b3ccf7aa
Submitter: Jenkins
Branch: stable/ocata

commit d99a06705489b1d63d72b0f2296d7236b3ccf7aa
Author: Sofer Athlan-Guyot <email address hidden>
Date: Thu Mar 23 12:10:48 2017 +0100

    [N->O] Fix wrong database connection for cell0 during upgrade.

    During upgrade the cell0 database has the connection pointing to

       mysql+pymysql://nova:c2cdagE8PyAbnpers3AD88Hge@10.0.0.19/nova_cell0?bind_address=10.0.0.20

    where 10.0.0.20 was the ip of the bootstrap node. This makes the
    nova-api fails on 2/3 node at the end of the
    major-upgrade-composable-steps.yaml step.

    We do have the right value in the hiera database so make sure we use
    it for cell0 creation and not the nova.conf file which hasn't been
    updated yet.

    Change-Id: I09775206cb8fc5e15934f7e4475506a7fe17271e
    Closes-Bug: #1675359
    (cherry picked from commit c9c3813b6a0811a262068d0aab28d0bd535be3e1)

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0b1 development milestone.

This issue was fixed in the openstack/tripleo-heat-templates 6.1.0 release.