kolla-ansible

mariadb_recovery didn't set restart policy when recreate mariadb container.

Bug #1851594 reported by Eddie Yen
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
Fix Released
High
Mark Goddard
kolla-ansible 10.0.0 "ussuri"
Rocky
Fix Released
High
Mark Goddard
kolla-ansible 7.2.0 "Rocky"
Stein
Fix Released
High
Mark Goddard
kolla-ansible 8.1.0 "Stein"
Train
Fix Released
High
Mark Goddard
kolla-ansible 9.0.0 "Train"
Ussuri
Fix Released
High
Mark Goddard
kolla-ansible 10.0.0 "ussuri"

Bug Description

In default, every Openstack container will set restart policy, and the default value is 'unless-stopped' if not change restart policy value inside all.yml

When doing mariadb_recovery process, the container will re-create and start new mariadb process. But we found that the re-created container didn't set restart policy when check mariadb container using docker inspect.

Expected result after mariadb_recovery:

"RestartPolicy": {
    "Name": "unless-stopped",
    "MaximumRetryCount": 0
},

Actual result after mariadb_recovery:

"RestartPolicy": {
    "Name": "",
    "MaximumRetryCount": 0
},

OS: Ubuntu 18.04
Docker version: 19.03.3
Kolla-Ansible version: stable/rocky
Docker image install type: source

Mark Goddard (mgoddard)
Changed in kolla-ansible:
status: New → Triaged
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (master)

Fix proposed to branch: master
Review: https://review.opendev.org/693329

Changed in kolla-ansible:
assignee: nobody → Mark Goddard (mgoddard)
status: Triaged → In Progress
Mark Goddard (mgoddard)
Changed in kolla-ansible:
milestone: 9.0.0 → none
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/693329
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=f979ae1f8ee182a8bc6fde4b23b95a460cb81488
Submitter: Zuul
Branch: master

commit f979ae1f8ee182a8bc6fde4b23b95a460cb81488
Author: Mark Goddard <email address hidden>
Date: Thu Nov 7 10:01:14 2019 +0000

    Fix restart policy after MariaDB recovery

    After performing a recovery of MariaDB, the mariadb containers are left
    without a restart policy. This leaves them unable to recover from the
    crash of a single galera node. There is another issue, in that the
    'master' node is left in a bootstrap configuration, with the
    --wsrep-new-cluster argument configured as BOOTSTRAP_ARGS.

    This change fixes these issues by removing the restart policy of 'no'
    from the 'slave' containers, and recreating the master container without
    the restart policy or bootstrap arguments.

    Change-Id: I36c875611931163ca2c29ae93b71d3af64cb197c
    Closes-Bug: #1851594

Changed in kolla-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/694550

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/train)

Reviewed: https://review.opendev.org/694550
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=0a56767caed7fe89150e55ca7b9d3f546b5b865f
Submitter: Zuul
Branch: stable/train

commit 0a56767caed7fe89150e55ca7b9d3f546b5b865f
Author: Mark Goddard <email address hidden>
Date: Thu Nov 7 10:01:14 2019 +0000

    Fix restart policy after MariaDB recovery

    After performing a recovery of MariaDB, the mariadb containers are left
    without a restart policy. This leaves them unable to recover from the
    crash of a single galera node. There is another issue, in that the
    'master' node is left in a bootstrap configuration, with the
    --wsrep-new-cluster argument configured as BOOTSTRAP_ARGS.

    This change fixes these issues by removing the restart policy of 'no'
    from the 'slave' containers, and recreating the master container without
    the restart policy or bootstrap arguments.

    Change-Id: I36c875611931163ca2c29ae93b71d3af64cb197c
    Closes-Bug: #1851594
    (cherry picked from commit f979ae1f8ee182a8bc6fde4b23b95a460cb81488)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/695289

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/695290

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/rocky)

Reviewed: https://review.opendev.org/695290
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=a4e1074fb2ae17c43864b22ac006e72702111ba4
Submitter: Zuul
Branch: stable/rocky

commit a4e1074fb2ae17c43864b22ac006e72702111ba4
Author: Mark Goddard <email address hidden>
Date: Thu Nov 7 10:01:14 2019 +0000

    Fix restart policy after MariaDB recovery

    After performing a recovery of MariaDB, the mariadb containers are left
    without a restart policy. This leaves them unable to recover from the
    crash of a single galera node. There is another issue, in that the
    'master' node is left in a bootstrap configuration, with the
    --wsrep-new-cluster argument configured as BOOTSTRAP_ARGS.

    This change fixes these issues by removing the restart policy of 'no'
    from the 'slave' containers, and recreating the master container without
    the restart policy or bootstrap arguments.

    Change-Id: I36c875611931163ca2c29ae93b71d3af64cb197c
    Closes-Bug: #1851594
    (cherry picked from commit f979ae1f8ee182a8bc6fde4b23b95a460cb81488)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/stein)

Reviewed: https://review.opendev.org/695289
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=64e6b9e8802e1454bd37dc986ec650ffe6d6b2ee
Submitter: Zuul
Branch: stable/stein

commit 64e6b9e8802e1454bd37dc986ec650ffe6d6b2ee
Author: Mark Goddard <email address hidden>
Date: Thu Nov 7 10:01:14 2019 +0000

    Fix restart policy after MariaDB recovery

    After performing a recovery of MariaDB, the mariadb containers are left
    without a restart policy. This leaves them unable to recover from the
    crash of a single galera node. There is another issue, in that the
    'master' node is left in a bootstrap configuration, with the
    --wsrep-new-cluster argument configured as BOOTSTRAP_ARGS.

    This change fixes these issues by removing the restart policy of 'no'
    from the 'slave' containers, and recreating the master container without
    the restart policy or bootstrap arguments.

    Change-Id: I36c875611931163ca2c29ae93b71d3af64cb197c
    Closes-Bug: #1851594
    (cherry picked from commit f979ae1f8ee182a8bc6fde4b23b95a460cb81488)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 9.0.0.0rc2

This issue was fixed in the openstack/kolla-ansible 9.0.0.0rc2 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 7.2.0

This issue was fixed in the openstack/kolla-ansible 7.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 8.1.0

This issue was fixed in the openstack/kolla-ansible 8.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.