Ocata -> Pike upgrade of an environment with ceph nodes failure caused by failing 'Check legacy Ceph hieradata' task on OSD nodes

Description of problem:
Ocata -> Pike upgrade of an environment with ceph nodes failure caused by failing 'Check legacy Ceph hieradata' task on OSD nodes

Version-Release number of selected component (if applicable):

How reproducible:
100%

Steps to Reproduce:
1. Deploy Ocata with 3 controller + 2 computes + 3 ceph nodes

timeout 100m openstack overcloud deploy \
--templates /usr/share/openstack-tripleo-heat-templates \
--stack overcloud \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
-e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml \
-e /home/stack/virt/internal.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation-v6.yaml \
-e /home/stack/virt/network/network-environment-v6.yaml \
-e /home/stack/virt/enable-tls.yaml \
-e /home/stack/virt/inject-trust-anchor.yaml \
-e /home/stack/virt/public_vip.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/tls-endpoints-public-ip.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
--log-file overcloud_deployment_81.log

Content of: /home/stack/virt/internal.yaml

parameter_defaults:
    CinderEnableIscsiBackend: false
    CinderEnableRbdBackend: true
    CinderEnableNfsBackend: false
    NovaEnableRbdBackend: true
    GlanceBackend: rbd
    CinderRbdPoolName: "volumes"
    NovaRbdPoolName: "vms"
    GlanceRbdPoolName: "images"
    ExtraConfig:
      ceph::profile::params::osd_pool_default_pg_num: 32
      ceph::profile::params::osd_pool_default_pgp_num: 32
      ceph::profile::params::osds:
       '/dev/vdb': {}

2. Run major upgrade composable steps:

openstack overcloud deploy \
--templates /usr/share/openstack-tripleo-heat-templates \
--stack overcloud \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
-e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml \
-e /home/stack/virt/internal.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation-v6.yaml \
-e /home/stack/virt/network/network-environment-v6.yaml \
-e /home/stack/virt/enable-tls.yaml \
-e /home/stack/virt/inject-trust-anchor.yaml \
-e /home/stack/virt/public_vip.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/tls-endpoints-public-ip.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps-docker.yaml \
-e /home/stack/ceph-ansible-env.yaml \
-e /home/stack/extraconfig_override.yaml \
-e /home/stack/docker-osp12.yaml \

Note the /home/stack/extraconfig_override.yaml content that overrides ExtraConfig:

parameter_defaults:
    ExtraConfig:
        ceph::profile::params::osd_pool_default_pgp_num: 32
        ceph::profile::params::osd_pool_default_pg_num: 32

Actual results:
Upgrade fails caused by failing 'Check legacy Ceph hieradata' task on OSD nodes

overcloud.AllNodesDeploySteps.CephStorageUpgrade_Step0.1:
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: a9b9c88b-b88c-4a5a-8bb2-30911c9540e1
  status: CREATE_FAILED
  status_reason: |
    Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |
    ...
    TASK [Gathering Facts] *********************************************************
    ok: [localhost]

    TASK [Check legacy Ceph hieradata] *********************************************
    fatal: [localhost]: FAILED! => {"changed": true, "cmd": "test \"nil\" == \"$(hiera -c /etc/puppet/hiera.yaml ceph::profile::params::osds)\"", "delta": "0:00:00.094349", "end": "2018-03-02 21:01:47.052440", "msg": "non-zero return code", "rc": 1, "start": "2018-03-02 21:01:46.958091", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
     to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/39bb3320-0bd6-4365-8f0a-a0ca7674142c_playbook.retry

    PLAY RECAP *********************************************************************
    localhost : ok=1 changed=0 unreachable=0 failed=1

    (truncated, view all with --long)
  deploy_stderr: |

overcloud.AllNodesDeploySteps.CephStorageUpgrade_Step0.0:
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: e83240df-2a7c-4ea9-b8b0-3edab4a0c4c9
  status: CREATE_FAILED
  status_reason: |
    CREATE aborted
  deploy_stdout: |
None
  deploy_stderr: |
None
overcloud.AllNodesDeploySteps.CephStorageUpgrade_Step0.2:
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: 15e04b1d-bdd3-405d-9440-61face8ef484
  status: CREATE_FAILED
  status_reason: |
    CREATE aborted
  deploy_stdout: |
    ...
    TASK [Gathering Facts] *********************************************************
    ok: [localhost]

    TASK [Check legacy Ceph hieradata] *********************************************
    fatal: [localhost]: FAILED! => {"changed": true, "cmd": "test \"nil\" == \"$(hiera -c /etc/puppet/hiera.yaml ceph::profile::params::osds)\"", "delta": "0:00:00.097766", "end": "2018-03-02 21:01:47.675402", "msg": "non-zero return code", "rc": 1, "start": "2018-03-02 21:01:47.577636", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
     to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/cead8171-a86d-4c4c-9597-a7d960de0170_playbook.retry

    PLAY RECAP *********************************************************************
    localhost : ok=1 changed=0 unreachable=0 faileHeat Stack update failed.
Heat Stack update failed.
d=1

    (truncated, view all with --long)
  deploy_stderr: |

Expected results:
Upgrade succeeds.

Additional info:

It looks that the ExtraConfig override didn't get applied on the nodes:

cat controller-0/etc/puppet/hieradata/extraconfig.json
{
    "ceph::profile::params::osd_pool_default_pg_num": 32,
    "ceph::profile::params::osd_pool_default_pgp_num": 32,
    "ceph::profile::params::osds": {
        "/dev/vdb": {}
    }
}

cat ceph-0/etc/puppet/hieradata/extraconfig.json
{
    "ceph::profile::params::osd_pool_default_pg_num": 32,
    "ceph::profile::params::osd_pool_default_pgp_num": 32,
    "ceph::profile::params::osds": {
        "/dev/vdb": {}
    }