40-hiera-datafiles takes over 20 seconds to run

We should replace this with something that's run directly via heat-config, apart from being slow this implementation has no error path in the event something goes wrong (like a full disk for example).

I actually noticed this myself recently. This one script is eating up something like 6 minutes of our overcloud deploy time, just to translate some JSON to YAML.

Unfortunately, while it's pretty easy to rewrite the inner loop as Python (and it does massively decrease the runtime), because we do some icky things with jq there I don't think we'll get the same output from the Python version and I'm not sure how to make it the same. IIUC, we're reading some JSON values without parsing them in jq and in Python it's going to parse the entire thing at once so we lose the ability to read the raw unparsed values.

Of course, this partial json model is bad and really confusing anyway, so ideally we would just stop passing magic values that we don't want to parse and properly escape them. I'm not sure how huge a task that would be though, and it's possible it might break custom templates in the field that are relying on this bad behavior. :-(

Ben, I was looking at this around the time you wrote the above comment. While I had a handle on what the script is doing, I wasn't sure I fully understand the "why". After reading your comment, I'm actually 100% certain I don't understand the "why". I'll think about it some more, but maybe if you get some time you could walk me through what the requirements are with respect to the conversions? I have some ideas that involve piping intermediate forms but I'd be wasting my time if I don't understand the requirements.

Fix proposed to branch: master
Review: https://review.openstack.org/384756

Fix proposed to branch: master
Review: https://review.openstack.org/384757

Fix proposed to branch: master
Review: https://review.openstack.org/385438

Reviewed: https://review.openstack.org/384756
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=9f8944111c280de877326471899be23caca9bb94
Submitter: Jenkins
Branch: master

commit 9f8944111c280de877326471899be23caca9bb94
Author: Dan Prince <email address hidden>
Date: Mon Oct 10 21:03:12 2016 -0400

    Split out hosts config deployment

    This patch moves the hosts configuration into its own deployment.
    It will continue to use os-apply-config as something that is
    required early on in the bootstrapping (it needs to be
    configured before puppet runs for example).

    The motivation here is so we can refactor all-nodes-config.yaml to use a
    new hiera hook that that avoids os-apply-config entirely.

    Change-Id: Ib3e4380f205358b27d22a1102b663cf300b1ed86
    Partial-bug: #1596373

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/388245

Reviewed: https://review.openstack.org/388245
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=232e62762eca2257d9e359e15cb3508c82c0c38d
Submitter: Jenkins
Branch: stable/newton

commit 232e62762eca2257d9e359e15cb3508c82c0c38d
Author: Dan Prince <email address hidden>
Date: Mon Oct 10 21:03:12 2016 -0400

    Split out hosts config deployment

    This patch moves the hosts configuration into its own deployment.
    It will continue to use os-apply-config as something that is
    required early on in the bootstrapping (it needs to be
    configured before puppet runs for example).

    The motivation here is so we can refactor all-nodes-config.yaml to use a
    new hiera hook that that avoids os-apply-config entirely.

    Change-Id: Ib3e4380f205358b27d22a1102b663cf300b1ed86
    Partial-bug: #1596373
    (cherry picked from commit 9f8944111c280de877326471899be23caca9bb94)

Reviewed: https://review.openstack.org/385438
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=39b5e5e953f8b01821b55507b2e36985cb3072a1
Submitter: Jenkins
Branch: master

commit 39b5e5e953f8b01821b55507b2e36985cb3072a1
Author: Dan Prince <email address hidden>
Date: Wed Oct 12 08:18:40 2016 -0400

    Add new hiera agent hook to overcloud images.

    This patch adds the package for the new heat agent hiera hook
    to the overcloud image. This is a move towards helping us optimize the
    way Hiera is configured on the overcloud nodes.

    Depends-On: I21639f6aadabf9e49f40d1bb0b1d0edcfc4dbc5e

    Change-Id: Ia1864933235152b7e899c4442534879f8e22240d
    Partial-bug: #1596373

Fix proposed to branch: master
Review: https://review.openstack.org/393258

Fix proposed to branch: master
Review: https://review.openstack.org/393262

Reviewed: https://review.openstack.org/393262
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=36aa652247a83c868a25622bce54cc2b0247dc06
Submitter: Jenkins
Branch: master

commit 36aa652247a83c868a25622bce54cc2b0247dc06
Author: Dan Prince <email address hidden>
Date: Thu Nov 3 07:53:46 2016 -0400

    Neutron L3 service cleanups for hiera json hook

    This patch resolves a few issues I noticed when porting our
    Neutron L3 service to support the new heat hiera agent hook (which
    uses Json instead of Yaml).

     - If NeutronExternalNetworkBridge is an emptry string '' Json was
       dropping the single quotes thus causing the bridge to get set
       incorrectly in the config file. To correct this we use a heat
       conditional to avoid setting the external bridge (the '' default
       is what we want in this case) if the bridge is an empty string.

    Change-Id: I5037cbde6b76a37a4c22c4616278420e9d759109
    Partial-bug: #1596373

Fix proposed to branch: master
Review: https://review.openstack.org/397974

Reviewed: https://review.openstack.org/393258
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=133edad1302d09c2ad27d548fac6c3a72eec9c2e
Submitter: Jenkins
Branch: master

commit 133edad1302d09c2ad27d548fac6c3a72eec9c2e
Author: Dan Prince <email address hidden>
Date: Thu Nov 3 07:44:17 2016 -0400

    Horizon service cleanups for hiera json hook

    This patch resolves a few issues I noticed when porting our
    Horizon service to support the new heat hiera agent hook (which
    uses Json instead of Yaml).

     -we only need to set django_debug if the string is non-empty. This
      should match previous behavior.

     -remove the duplicated NeutronMechanismDrivers setting. This is already
      managed in the neutron services and shouldn't be set here.

    Change-Id: I473e110bb9b14cb8f57d41c4fc398871548726b0
    Partial-bug: #1596373

Reviewed: https://review.openstack.org/397974
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=8ab22a95dafcb19559524bb06f140214dceee501
Submitter: Jenkins
Branch: master

commit 8ab22a95dafcb19559524bb06f140214dceee501
Author: Dan Prince <email address hidden>
Date: Tue Nov 15 17:12:37 2016 -0500

    Nova base cleanups for hiera json hook

    This patch resolves an issue with nova-base.yaml that prevents
    it from working with the new heat hiera agent hook (which
    uses Json instead of Yaml).

    It updates the service so that we only set the upgrade level if it
    is not an empty string.

    Partial-bug: #1596373

    Change-Id: I595f2e16c33a6f935c7ca8935fec445d19c7b8f3

Reviewed: https://review.openstack.org/384757
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=787685101179f13f6074901070cf307bd9bb0731
Submitter: Jenkins
Branch: master

commit 787685101179f13f6074901070cf307bd9bb0731
Author: Dan Prince <email address hidden>
Date: Mon Oct 10 21:09:01 2016 -0400

    Hiera optimization: use a new hiera hook

    This patch optimizes how we deploy hiera by using a new
    heat hook specifically designed to help compose hiera
    within heat templates. As part of this change:

     - we update all the 'hiera' software configurations to set the group to hiera
       instead of os-apply-config.

     - The new format uses JSON instead of YAML. The hook actually writes
       out the hiera JSON directly so no conversion takes place. Arrays,
       Strings, Booleans all stay in their native formats. As such we can avoid
       having to do many of the awkward string and list conversions in t-h-t to
       support the previous YAML formatting.

     - The new hook prefers JSON over YAML so upgrading users will have the
       new files prefered. (we will post a cleanup routine for the old files
       soon but this isn't a new behavior, JSON is now simply prefered.)

     - A lot of services required edits to account for default settings that
       worked in YAML that no longer work correctly in the native JSON
       format. In almost all these cases I think the resulting codes looks
       cleaner and is more explicit with regards to what is getting
       configured in hiera on the actual nodes.

    Depends-On: I6a383b1ad4ec29458569763bd3f56fd3f2bd726b
    Closes-bug: #1596373

    Change-Id: Ibe7e2044e200e2c947223286fdf4fd5bcf98c2e1

This issue was fixed in the openstack/tripleo-heat-templates 6.0.0.0b2 development milestone.