trck the enablement of deep_compare with puppet-pacemaker on queens and rocky

Bug #1823849 reported by Michele Baldessari
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Triaged
Medium
Michele Baldessari

Bug Description

The deep comparison of pacemaker resources via puppet pacemaker has been enabled in rocky only.

There is still an optimization that we need to fix, namely that during minor update we potentially restart a resource even though we will stop the cluster on the node.

This bug is to track the following things:
a) enablement of deep_compare by default in queens
https://review.openstack.org/#/c/581416/

b) backport of potential minor update double restart in queens/rocky
https://review.openstack.org/#/c/635725/
https://review.openstack.org/#/c/635731/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/651110

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/rocky)

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/651114

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/651115

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/651181

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/651182

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/rocky)

Reviewed: https://review.openstack.org/651114
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=fb77857f7bf60d25dcde2f12eb7fdbcde327b61c
Submitter: Zuul
Branch: stable/rocky

commit fb77857f7bf60d25dcde2f12eb7fdbcde327b61c
Author: Michele Baldessari <email address hidden>
Date: Fri Feb 8 10:38:10 2019 +0100

    Be able to know when we are running inside a minor update workflow

    With this change we add an ansible variable called
    'tripleo_minor_update' set to true only during the update_steps_playbook
    which get run during a minor update.
    Then inside common/deploy-steps-tasks when starting containers with
    paunch we export this 'tripleo_minor_update' ansible variable and
    push it inside the 'TRIPLEO_MINOR_UPDATE' environment variable.

    Inside change Id1d671506d3ec827bc311b47d9363952e1239ce3 we will then
    use the env variable and export it to the restart_bundles in order
    to detect if we're inside a minor update workflow (as opposed to
    a redeploy - aka stack update). The testing that has been done is
    described in the above change.

    Co-Authored-By: Damien Ciabrini <email address hidden>

    Related-Bug: #1823849

    NB: Cherry-pick not 100% clean due to missing Ia630f08f553bd53656c76e5c8059f15d314a17c0

    Change-Id: Ib3562adbd83f7162c2aeb450329b7cc4ab200fc2
    (cherry picked from commit 7133394c35219966fea301bc779f0981808e38e6)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/651115
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=a39f0c56438a9a588685fa9a9016f36c43901c25
Submitter: Zuul
Branch: stable/rocky

commit a39f0c56438a9a588685fa9a9016f36c43901c25
Author: Michele Baldessari <email address hidden>
Date: Fri Feb 8 10:56:53 2019 +0100

    Do not restart bundles during a minor update

    It makes no sense since we are going to stop the cluster on the node
    anyway. With change Ib3562adbd83f7162c2aeb450329b7cc4ab200fc2 we
    inject the TRIPLEO_MINOR_UDPATE variable into the update playbooks
    and we also push it (by default to false) inside the paunch start
    container step.

    By exporting the variable TRIPLEO_MINOR_UPDATE without a value we
    tell docker (and podman) to inherit this value from the host.
    When the env variable is set to true we avoid calling pcs inside
    the containers to restart the bundles because at that point we
    know that restarting the HA container would make no sense.

    We tested the changes as follows:
    A) Ran a minor update (openstack overcloud update prepare + openstack
    overcloud update run --roles Controller) and observed that there were no
    extra pcmk-managed bundles restarts (i.e. even when paunch decided to
    invoke the <service>_restart_bundle it was a noop because
    TRIPLEO_MINOR_UDPATE was 'true')
    B) We ran a re-deploy without any changes and observed that no restart
    of pcmk-managed bundles took place
    C) We ran a re-deploy and forced some configuration changes and observed
    that TRIPLEO_MINOR_UDPATE was false and that the
    <service>_restart_bundle was run by paunch and that it restarted the
    bundle that had the config changed.

    Co-Authored-By: Damien Ciabrini <email address hidden>

    Related-Bug: #1823849

    Change-Id: Id1d671506d3ec827bc311b47d9363952e1239ce3
    (cherry picked from commit e6ab4856d7982b208a16443dc3e7e0f3fd136113)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/queens)

Reviewed: https://review.openstack.org/651181
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=79eddd1552099ed212fb06b30490521935015efe
Submitter: Zuul
Branch: stable/queens

commit 79eddd1552099ed212fb06b30490521935015efe
Author: Michele Baldessari <email address hidden>
Date: Tue Apr 9 12:04:44 2019 +0200

    Be able to know when we are running inside a minor update workflow

    With this change we add an ansible variable called
    'tripleo_minor_update' set to true only during the update_steps_playbook
    which get run during a minor update.
    Then inside common/deploy-steps-tasks when starting containers with
    paunch we export this 'tripleo_minor_update' ansible variable and
    push it inside the 'TRIPLEO_MINOR_UPDATE' environment variable.

    Inside change Id1d671506d3ec827bc311b47d9363952e1239ce3 we will then
    use the env variable and export it to the restart_bundles in order
    to detect if we're inside a minor update workflow (as opposed to
    a redeploy - aka stack update). The testing that has been done is
    described in the above change.

    Co-Authored-By: Damien Ciabrini <email address hidden>

    Related-Bug: #1823849

    NB: Cherry-pick not 100% clean due to missing Ia630f08f553bd53656c76e5c8059f15d314a17c0
        and other missing reviews

    Change-Id: Ib3562adbd83f7162c2aeb450329b7cc4ab200fc2
    (cherry picked from commit 7133394c35219966fea301bc779f0981808e38e6)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/651182
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=dc9a67ced5f8cca1bc26fe8ee0fec610f197db7e
Submitter: Zuul
Branch: stable/queens

commit dc9a67ced5f8cca1bc26fe8ee0fec610f197db7e
Author: Michele Baldessari <email address hidden>
Date: Fri Feb 8 10:56:53 2019 +0100

    Do not restart bundles during a minor update

    It makes no sense since we are going to stop the cluster on the node
    anyway. With change Ib3562adbd83f7162c2aeb450329b7cc4ab200fc2 we
    inject the TRIPLEO_MINOR_UDPATE variable into the update playbooks
    and we also push it (by default to false) inside the paunch start
    container step.

    By exporting the variable TRIPLEO_MINOR_UPDATE without a value we
    tell docker (and podman) to inherit this value from the host.
    When the env variable is set to true we avoid calling pcs inside
    the containers to restart the bundles because at that point we
    know that restarting the HA container would make no sense.

    We tested the changes as follows:
    A) Ran a minor update (openstack overcloud update prepare + openstack
    overcloud update run --roles Controller) and observed that there were no
    extra pcmk-managed bundles restarts (i.e. even when paunch decided to
    invoke the <service>_restart_bundle it was a noop because
    TRIPLEO_MINOR_UDPATE was 'true')
    B) We ran a re-deploy without any changes and observed that no restart
    of pcmk-managed bundles took place
    C) We ran a re-deploy and forced some configuration changes and observed
    that TRIPLEO_MINOR_UDPATE was false and that the
    <service>_restart_bundle was run by paunch and that it restarted the
    bundle that had the config changed.

    Co-Authored-By: Damien Ciabrini <email address hidden>

    Related-Bug: #1823849

    NB: not 100% clean cherry-pick due to {notify,rpc}-rabbitmq.yaml files
    missing

    Change-Id: Id1d671506d3ec827bc311b47d9363952e1239ce3
    (cherry picked from commit e6ab4856d7982b208a16443dc3e7e0f3fd136113)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/651110
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=2a734381fc9ff3f1e1e369d20f2bd4fdb62c1237
Submitter: Zuul
Branch: stable/queens

commit 2a734381fc9ff3f1e1e369d20f2bd4fdb62c1237
Author: Michele Baldessari <email address hidden>
Date: Tue Jul 10 18:14:38 2018 +0200

    Enable deep_compare of pcmk resources by default

    By settings these hiera keys we will actually do a deep comparison
    of all pacemaker resources and update them accordingly.

    This means that rerunning a deploy against an existing overcloud
    will actually detect if a pacemaker resource needs updating and
    will update the definition and restart of said resource. The code
    will wait for the resource to be up (aka the cluster to settle
    after an update).

    Related-Bug: #1823849

    Depends-On: I62d5662327333e17e1c32a029695b3d3a904ca10
    Change-Id: I0b9ed429e6b1bd4a62b4462bf6312782f29b7d6b
    (cherry picked from commit 60d75f19e5165d784d0d46c6addb81fe4a8b2876)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.