kolla-ansible

keepalived not upgraded when going from train to ussuri

  1. Series ussuri
  2. Bug #1928362
Bug #1928362 reported by Will Szumski
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
Invalid
Undecided
Unassigned
Ussuri
Fix Committed
Medium
Unassigned

Bug Description

I did an upgrade from train to ussuri via kayobe with:

kayobe overcloud service upgrade

I saw the old train container hanging around:

[centos@kayobe-aio ~]$ sudo docker ps | grep 9.3.1.6-1
53219d352c1f 10.60.150.1:4000/rse-cambridge/centos-binary-prometheus-libvirt-exporter:9.3.1.6-1-centos8 "dumb-init --single-…" 2 days ago Up 2 days prometheus_libvirt_exporter
d78ee6384ee3 10.60.150.1:4000/rse-cambridge/centos-binary-prometheus-jiralert:9.3.1.6-1-centos8 "dumb-init --single-…" 2 days ago Up 2 days prometheus_jiralert
f3dba9981adb 10.60.150.1:4000/rse-cambridge/centos-binary-prometheus-mtail:9.3.1.6-1-centos8 "dumb-init --single-…" 2 days ago Up 2 days prometheus_mtail
36d1804256c0 10.60.150.1:4000/rse-cambridge/centos-binary-keepalived:9.3.1.6-1-centos8 "dumb-init --single-…" 2 days ago Up 2 days

Running `kayobe overcloud service deploy --skip-prechecks -t none -kt haproxy` upgraded the container. I thought this looked more like a kolla-ansible bug than a kayobe bug, so reporting here.

I am running a few patches on top of current stable/ussuri. So it is entirely possible that I broke something too.

Revision history for this message
Will Szumski (willjs) wrote :

I just checked an I don't have any notable changes to the haproxy rule, so assume this bug is upstream.

Revision history for this message
Will Szumski (willjs) wrote :

Noticed that we now call check-containers as part of upgrade, so this is probably no longer an issue on master. In ussuri it seems like to rely on a change in config.yml to restart the container. When there are no changes to the config files it doesn't get triggered. I'm running on an environment with a single host, but it seems like backup keepalived instances would be restarted due to:

https://github.com/openstack/kolla-ansible/blob/stable/ussuri/ansible/roles/haproxy/tasks/upgrade.yml#L15

Mark Goddard (mgoddard)
Changed in kolla-ansible:
importance: Undecided → Medium
status: New → Invalid
importance: Medium → Undecided
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/791694

Revision history for this message
Pierre Riteau (priteau) wrote :

I confirm that I am seeing this on another deployment. Backup keepalived containers got updated, but not the primary one.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/791694
Committed: https://opendev.org/openstack/kolla-ansible/commit/2c728619301240c6f52894a4d0b48d0d888eaa81
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 2c728619301240c6f52894a4d0b48d0d888eaa81
Author: Mark Goddard <email address hidden>
Date: Mon May 17 09:32:40 2021 +0100

    Ensure keepalived is upgraded

    Ussuri & earlier only

    After upgrading from Train to Ussuri, if the keepalived configuration is
    unchanged, it is possible that the primary keepalived container will not
    be upgraded. This happens because we do not import check-containers.yml
    in upgrade.yml, meaning that the 'Restart keepalived container' handler
    does not fire. This change fixes the issue.

    Closes-Bug: #1928362

    Change-Id: I56775f1c0a8849c10ad5181cde6b50b2694a0512

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/792517

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/train)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/792517
Committed: https://opendev.org/openstack/kolla-ansible/commit/26d310eea3f1c83dc4eb8f5806f9aad55d4a123e
Submitter: "Zuul (22348)"
Branch: stable/train

commit 26d310eea3f1c83dc4eb8f5806f9aad55d4a123e
Author: Mark Goddard <email address hidden>
Date: Mon May 17 09:32:40 2021 +0100

    Ensure keepalived is upgraded

    Ussuri & earlier only

    After upgrading from Train to Ussuri, if the keepalived configuration is
    unchanged, it is possible that the primary keepalived container will not
    be upgraded. This happens because we do not import check-containers.yml
    in upgrade.yml, meaning that the 'Restart keepalived container' handler
    does not fire. This change fixes the issue.

    Closes-Bug: #1928362

    Change-Id: I56775f1c0a8849c10ad5181cde6b50b2694a0512
    (cherry picked from commit 2c728619301240c6f52894a4d0b48d0d888eaa81)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 9.3.2

This issue was fixed in the openstack/kolla-ansible 9.3.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 10.3.0

This issue was fixed in the openstack/kolla-ansible 10.3.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.