tripleo

Invalid facts on external upgrade run

Bug #1900836 reported by Sergii Golovatiuk on 2020-10-21

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Confirmed	High	Kevin Carter	tripleo victoria-3 "tripleo victoria"

Bug Description

Analyzing CI failure [1] I have found the following situation.

The job tries to upgrade from rocky (which is docker based) to stein (which is podman based)

Upgrade steps were run executing tripleo-docker-rm, removing docker as part of upgrade to podman.

After upgrade steps "External upgrade" steps where processed. On External upgrade step 1, "Online data migration for Cinder" happened which is described as

     external_upgrade_tasks:
        - when: step|int == 1
          block:
            - name: Online data migration for Cinder
              command: "{{ container_cli }} exec cinder_api cinder-manage db online_data_migrations"
              delegate_to: "{{ groups['cinder_api'][0] }}"
              become: true
              tags:
                - online_upgrade
                - online_upgrade_cinder

So, container_cli should be resolved as podman but it's not as container_cli fact was not updated. Thus, ansible considered that container_cli was still docker.

Steps to reproduce
==================
Deploy standalone rocky
Upgrade to standalone stein

Expected result
===============
Upgraded environment

Actual result
=============
Failed standalone upgrade due to wrong fact.

[1] https://api.us-east.open-edge.io:8080/swift/v1/AUTH_e02c11e4e2c24efc98022353c88ab506/zuul_opendev_logs_8c2/752938/1/check/tripleo-ci-centos-7-standalone-upgrade-stein/8c2ea1e/logs/undercloud/home/zuul/standalone_upgrade.log

Tags:

Revision history for this message

Sergii Golovatiuk (sgolovatiuk) wrote on 2020-10-21:

I guess in tasks/tripleo_docker_purge.yml we should rather do

- name: clearing facts
meta: clear_facts

instead of

     - name: Refresh hardware facts
       setup:
         gather_subset:
           - hardware

Changed in tripleo:
assignee:	nobody → Kevin Carter (kevin-carter)
importance:	Undecided → High
status:	New → Confirmed

Revision history for this message

Kevin Carter (kevin-carter) wrote on 2020-10-22:

+1 on to using clear facts. While this would update the facts globally for the playbook context, forcing everything to refresh, it would ensure we have the most up-to-date information when performing this action which I think is worth the "cost" of regathering facts.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-10-22: Fix proposed to tripleo-heat-templates (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/759303

Bogdan Dobrelya (bogdando) on 2020-10-23

Changed in tripleo:
milestone:	none → victoria-3

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-11-03: Fix merged to tripleo-heat-templates (stable/stein)

Reviewed: https://review.opendev.org/759303
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=7e937946005191a9362f61f0ab91b317a267bcd6
Submitter: Zuul
Branch: stable/stein

commit 7e937946005191a9362f61f0ab91b317a267bcd6
Author: Kevin Carter <email address hidden>
Date: Thu Oct 22 12:09:25 2020 -0500

Add clear facts to the start of all major playbooks

    This change will ensure that any and all ansible facts are cleared
    when starting any and all core framework interactions. By clearning
    the cached facts we'll ensure we're not running into legacy or
    otherwise outdated fact information.

This change is adapted from master [0].

[0] https://github.com/openstack/tripleo-heat-templates/commit/1f9942fd755a168f8a9b75afa09e8af7027d61d4

    Change-Id: I2410f86db063ef3c037baf2bc716075bda40f11c
    Closes-Bug: 1900836
    Signed-off-by: Kevin Carter <email address hidden>

tags:

added: in-stable-stein

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-07-23: Fix included in openstack/tripleo-heat-templates stein-eol

This issue was fixed in the openstack/tripleo-heat-templates stein-eol release.