tripleo

Centos9 Wallaby Tripleo OVB jobs are failing on provisioning

Bug #2030321 reported by Sandeep Yadav
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Unassigned
tripleo antelope-1

Bug Description

Description:

Centos9 Wallaby Tripleo OVB jobs are failing on provisioning

Issue started on 29th July as per build history:

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby&skip=0

Failure: Doesn't move after grub

https://logserver.rdoproject.org/09/49509/1/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/7c4ed1a/logs/baremetal_1_66418_1-console.log

~~~
[2J[01;01H[=3h[2J[01;01H[0m[35m[40m[0m[37m[40mBdsDxe: loading Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x2,0x7)/Pci(0x0,0x0)
BdsDxe: starting Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x2,0x7)/Pci(0x0,0x0)
System BootOrder not found. Initializing defaults.
Creating boot entry "Boot0003" with label "CentOS Linux" for file "\EFI\centos\shimx64.efi"
.
.
The highlighted entry will be executed automatically in 0s. [5;78H[0m[30m[40m[2J[01;01H[0m[37m[40m[?25h[H[J[1;1H[H[J[1;1H[0m[30m[40m[2J[01;01H[0m[37m[40m Booting `CentOS Stream (5.14.0-344.el9.x86_64) 9'Booting `CentOS Stream (5.14.0-344.el9.x86_64) 9'
~~~

Tags: ci
Revision history for this message
Sandeep Yadav (sandeepyadav93) wrote :

diff between last good (28th) vs first failing(31st) log: https://www.diffchecker.com/vZKGLmB9/

Revision history for this message
Sandeep Yadav (sandeepyadav93) wrote :

Excluding just grub2 helped in fixing overcloud boot issue which we were facing earlier in ovb jobs.

Exclude Patch:
[1] https://review.opendev.org/c/openstack/tripleo-quickstart/+/890579/6/config/release/tripleo-ci/CentOS-9/promotion-testing-hash-wallaby.yml
    exclude:
      - grub2*2.06-67*

Testproject green result with grub2 exclude:

https://review.rdoproject.org/r/c/testproject/+/39357/58#message-667d4aa1e0004a22d8a042322b87121f7ca91bd6

OpenStack Infra (hudson-openstack)
Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-ci (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-ci/+/890345
Committed: https://opendev.org/openstack/tripleo-ci/commit/ffc65c4be05d3590a37076ba851dc4b0184ffb9e
Submitter: "Zuul (22348)"
Branch: master

commit ffc65c4be05d3590a37076ba851dc4b0184ffb9e
Author: Ronelle Landy <email address hidden>
Date: Wed Aug 2 11:16:12 2023 -0400

    Use pinned centos-9 cloud image

    The latest centos-9 cloud image fails
    node provision due to an issue with
    CentOS kernel 5.14.0-344.el9.x86_64.
    This patch fixes the cloud image to
    the last known good version.

    Related-Bug: #2030321

    Change-Id: Ie752f4cb9d798599e5d4d5b6154df7d71d5fa32a

Revision history for this message
Sandeep Yadav (sandeepyadav93) wrote :

Here is the grub2 bug[1], As per the latest on the bug - Centos team is pinning last good known grub in centos compose.

 [1] https://bugzilla.redhat.com/show_bug.cgi?id=2228422

 [2] https://bugzilla.redhat.com/show_bug.cgi?id=2227722#c10

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-quickstart (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/890579
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/9109140002a33e89dc83bcb7b808361cda0f3e61
Submitter: "Zuul (22348)"
Branch: master

commit 9109140002a33e89dc83bcb7b808361cda0f3e61
Author: Sandeep Yadav <email address hidden>
Date: Sat Aug 5 11:10:27 2023 +0530

    Temporarily Exclude latest Grub2

    With the latest grub2*2.06-67*, ovb jobs
    overcloud nodes can't boot, details in
    Closes-Bug.

    Excluding grub2 .67 version till we figure out what's
    the issue and get a resolution

    Closes-Bug: #2030321

    Change-Id: I1005e240f298003900b318bc79cf30194a59b109

Changed in tripleo:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.