tripleo

centos-8 multinode-oooq-container-updates fails during converge "Failed containers: keystone_bootstrap"

Bug #1932261 reported by Marios Andreou
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Alex Schultz
tripleo xena-1

Bug Description

At [1][2][3] the tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates is failing during the converge step with trace like

 2021-06-17 01:55:21 | 2021-06-17 01:55:21.379561 | | WARNING | Failure running exec 'keystone_bootstrap'. rc=255, stdout=, stderr=Error: can only create exec sessions on running containers: container state improper
 2021-06-17 01:55:21 | 2021-06-17 01:55:21.381411 | bc764e10-1505-d63c-eddc-0000000013f6 | FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_3 | centos-8-stream-rax-ord-0025151891 | error={"changed": false, "msg": "Failed containers: keystone_bootstrap"}

This has started happening within the last 12 hours or so [4], so it must be something that merged very recently (and where we aren't gating with the update job, e.g. tripleo-ansible but I have not yet confirmed this). This is a master gate blocker. Last successful runs from yesterday there [5]

[1] https://98f884550bb898129005-dc55dfdaf5877a4d04198310cb0fd46a.ssl.cf1.rackcdn.com/796712/1/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/36713ad/logs/undercloud/home/zuul/overcloud_update_converge.log
[2] https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_467/778915/40/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/467d23f/logs/undercloud/home/zuul/overcloud_update_converge.log
[3] https://0cbbb3ecadec96097039-d3b9488846fd47e876cca24d70452d35.ssl.cf1.rackcdn.com/796531/2/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/bfe3551/logs/undercloud/home/zuul/overcloud_update_converge.log
[4] https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates
[5] https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates&result=SUCCESS

Marios Andreou (marios-b)
Changed in tripleo:
importance: Undecided → Critical
tags: added: alert
Revision history for this message
Marios Andreou (marios-b) wrote :

Not confirmed yet but I suspect this may have started with https://review.opendev.org/c/openstack/tripleo-ansible/+/791317 Container manage module

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-ansible (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/796805

Revision history for this message
Marios Andreou (marios-b) wrote :

posted https://review.opendev.org/c/openstack/tripleo-ansible/+/796805 to wire up the minor update job to run in tripleo-ansible

Revision history for this message
Marios Andreou (marios-b) wrote :

also adding the promotion-blocker tag so it will be tracked as CIX it isn't currently on the board

tags: added: promotion-blocker
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/796857

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/796857
Committed: https://opendev.org/openstack/tripleo-ansible/commit/6a24ca732529b282315da114273f6f79001d0a44
Submitter: "Zuul (22348)"
Branch: master

commit 6a24ca732529b282315da114273f6f79001d0a44
Author: Alex Schultz <email address hidden>
Date: Thu Jun 17 13:28:41 2021 +0000

    Revert "Container manage module"

    This reverts commit 22bbe1311094be600b9fed426f1ee0a983b8e111.

    This appears to have broken updates. Let's revert for now.

    Change-Id: I680b38643151a2a16205e15edd3ea8703214a2d8
    Related-Bug: #1932261

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/796805
Committed: https://opendev.org/openstack/tripleo-ansible/commit/e8570786776612deac68e332b51331fac3a755e1
Submitter: "Zuul (22348)"
Branch: master

commit e8570786776612deac68e332b51331fac3a755e1
Author: Marios Andreou <email address hidden>
Date: Thu Jun 17 12:07:41 2021 +0300

    Add minor update job to tripleo-ansible zuul layout

    This wires up the master upgrades template so we will gate on
    the minor update job in tripleo-ansible to prevent issues like
    related-bug.

    Related-Bug: 1932261
    Change-Id: I3711d0ef8cf8daecde2f9593bf3b8214ddc647ca

Bogdan Dobrelya (bogdando)
Changed in tripleo:
status: Triaged → Fix Released
assignee: nobody → Alex Schultz (alex-schultz)
Revision history for this message
Marios Andreou (marios-b) wrote :

revert merged and the job is back to green https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates thanks for fast actions to clear this

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.