tripleo-ci-centos-9-undercloud-upgrade failing consistently while updating all packages

Bug #1988397 reported by Pooja Jadhav
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Triaged
Critical
Unassigned

Bug Description

tripleo-ci-centos-9-undercloud-upgrade failing consistently while updating all packages with below traceback :

2022-09-01 07:06:25 | 2022-09-01 07:06:25.280385 | fa163ee2-35df-fd17-55b8-000000000550 | TASK | Update all packages
2022-09-01 07:06:26 | 2022-09-01 07:06:26.573497 | fa163ee2-35df-fd17-55b8-000000000550 | FATAL | Update all packages | undercloud | error={"changed": false, "failures": [], "msg": "Depsolve Error occurred: \n Problem: package os-net-config-15.2.1-0.20220629114404.6505f24.el9.noarch requires NetworkManager-ovs, but none of the providers can be installed\n - package NetworkManager-ovs-1:1.39.10-1.el9.x86_64 requires NetworkManager(x86-64) = 1:1.39.10-1.el9, but none of the providers can be installed\n - package NetworkManager-ovs-1:1.39.12-1.el9.x86_64 requires NetworkManager(x86-64) = 1:1.39.12-1.el9, but none of the providers can be installed\n - package NetworkManager-ovs-1:1.39.6-1.el9.x86_64 requires NetworkManager(x86-64) = 1:1.39.6-1.el9, but none of the providers can be installed\n - package NetworkManager-ovs-1:1.39.7-2.el9.x86_64 requires NetworkManager(x86-64) = 1:1.39.7-2.el9, but none of the providers can be installed\n - package NetworkManager-ovs-1:1.39.90-1.el9.x86_64 requires NetworkManager(x86-64) = 1:1.39.90-1.el9, but none of the providers can be installed\n - cannot install both NetworkManager-1:1.39.10-1.el9.x86_64 and NetworkManager-1:1.40.0-1.el9.x86_64\n - cannot install both NetworkManager-1:1.39.12-1.el9.x86_64 and NetworkManager-1:1.40.0-1.el9.x86_64\n - cannot install both NetworkManager-1:1.39.6-1.el9.x86_64 and NetworkManager-1:1.40.0-1.el9.x86_64\n - cannot install both NetworkManager-1:1.39.7-2.el9.x86_64 and NetworkManager-1:1.40.0-1.el9.x86_64\n - cannot install both NetworkManager-1:1.39.90-1.el9.x86_64 and NetworkManager-1:1.40.0-1.el9.x86_64\n - cannot install the best update candidate for package os-net-config-14.2.1-0.20220624004921.755f151.el9.noarch\n - problem with installed package NetworkManager-1:1.40.0-1.el9.x86_64", "rc": 1, "results": []}
2022-09-01 07:06:26 | 2022-09-01 07:06:26.575110 | fa163ee2-35df-fd17-55b8-000000000550 | TIMING | Update all packages | undercloud | 0:01:08.655448 | 1.29s

Detailed Logs :

https://32f730b73845cbdfd0fc-4cf4bb096d4774f141f32d30757edf40.ssl.cf1.rackcdn.com/853411/2/check/tripleo-ci-centos-9-undercloud-upgrade/b6a6d75/logs/undercloud/home/zuul/undercloud_upgrade.log

https://ef8e3d7fa760b4ab7420-4082203e55009ffc8095c87eb70265ba.ssl.cf5.rackcdn.com/854602/1/gate/tripleo-ci-centos-9-undercloud-upgrade/dc4a2fd/logs/undercloud/home/zuul/undercloud_upgrade.log

Tags: ci
Revision history for this message
Jakob Meng (jm1337) wrote :

      <ysandeep> we still have mirror issues:
      <ysandeep> job have .40 version installed
      <ysandeep> https://32f730b73845cbdfd0fc-4cf4bb096d4774f141f32d30757edf40.ssl.cf1.rackcdn.com/853411/2/check/tripleo-ci-centos-9-undercloud-upgrade/b6a6d75/logs/undercloud/var/log/extra/package-list-installed.txt
      <ysandeep> ~~~
      <ysandeep> NetworkManager.x86_64 1:1.40.0-1.el9 @baseos
      <ysandeep> NetworkManager-libnm.x86_64 1:1.40.0-1.el9 @baseos
      <ysandeep> ~~~
      <ysandeep> I don't see .40 rpms for NetworkManager on rackspace mirror: http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/
      <ykarel> facebook mirror looks good
      <ykarel> rackspace one not updated since 10 hours
      <ykarel> http://dfw.mirror.rackspace.com/centos-stream/timestamp.txt
      <jm1> ysandeep, ykarel: and neither on http://mirror.mtl01.iweb.opendev.org/centos-stream/9-stream/AppStream/x86_64/os/Packages/
      <ykarel> yes ^ syncs from mirror which is rax currently so should be same
      <ykarel> http://mirror.mtl01.iweb.opendev.org/centos-stream/timestamp.txt
      <ykarel> 2022-09-01T06:08:45,769043757+00:00

Revision history for this message
yatin (yatinkarel) wrote :

Pushed https://review.opendev.org/c/opendev/system-config/+/855457 to switch back to facebook mirror as that's up to date currently.

Since CI jobs use infra mirrors, it makes sense nodepool images also use infra mirrors to avoid these issues issues in future.

While discussing it on #opendev[1] it's found that CentOS images not using infra mirrors like some non centos image using(via DIB_DISTRIBUTION_MIRROR)[1], so need to make CentOS images such that infra mirror repos get's used, currently centos repos are getting used[3]

[1] https://meetings.opendev.org/irclogs/%23opendev/%23opendev.2022-09-01.log.html#t2022-09-01T12:09:25
[2] https://opendev.org/openstack/project-config/src/branch/master/nodepool/nodepool.yaml#L225
[3] https://github.com/openstack/diskimage-builder/blob/master/diskimage_builder/elements/centos-minimal/yum.repos.d/9-stream

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.