tripleo-ci-centos-9-undercloud-upgrade - cannot install both NetworkManager-1:1.39.5-1.el9.x86_64 and NetworkManager-1:1.39.12

Bug #1984175 reported by Marios Andreou
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Undecided
Unassigned

Bug Description

At [1][2][3] the tripleo-ci-centos-9-undercloud-upgrade fails during the undercloud upgrade and specifically package conflict when upgrading packages. Trace looks like:

        * 2022-08-10 10:02:29 | 2022-08-10 10:02:29.642254 | bc764e04-48fc-91a2-43a1-000000000550 | FATAL | Update all packages | undercloud | error={"changed": false, "failures": [], "msg": "Depsolve Error occurred: \n Problem: package os-net-config-15.2.1-0.20220629114404.6505f24.el9.noarch requires NetworkManager-ovs, but none of the providers can be installed\n - package NetworkManager-ovs-1:1.39.10-1.el9.x86_64 requires NetworkManager(x86-64) = 1:1.39.10-1.el9, but none of the providers can be installed\n - package NetworkManager-ovs-1:1.39.3-1.el9.x86_64 requires NetworkManager(x86-64) = 1:1.39.3-1.el9, but none of the providers can be installed\n - package NetworkManager-ovs-1:1.39.5-1.el9.x86_64 requires NetworkManager(x86-64) = 1:1.39.5-1.el9, but none of the providers can be installed\n - package NetworkManager-ovs-1:1.39.6-1.el9.x86_64 requires NetworkManager(x86-64) = 1:1.39.6-1.el9, but none of the providers can be installed\n - package NetworkManager-ovs-1:1.39.7-2.el9.x86_64 requires NetworkManager(x86-64) = 1:1.39.7-2.el9, but none of the providers can be installed\n - cannot install both NetworkManager-1:1.39.10-1.el9.x86_64 and NetworkManager-1:1.39.12-1.el9.x86_64\n - cannot install both NetworkManager-1:1.39.3-1.el9.x86_64 and NetworkManager-1:1.39.12-1.el9.x86_64\n - cannot install both NetworkManager-1:1.39.5-1.el9.x86_64 and NetworkManager-1:1.39.12-1.el9.x86_64\n - cannot install both NetworkManager-1:1.39.6-1.el9.x86_64 and NetworkManager-1:1.39.12-1.el9.x86_64\n - cannot install both NetworkManager-1:1.39.7-2.el9.x86_64 and NetworkManager-1:1.39.12-1.el9.x86_64\n - cannot install the best update candidate for package os-net-config-14.2.1-0.20220624004921.755f151.el9.noarch\n - problem with installed package NetworkManager-1:1.39.12-1.el9.x86_64", "rc": 1, "results": []}

This blocks the master gate across most tripleo repos - I can see tripleo-heat-templates, tripleo-common, tripleo-ansible, tripleo-quickstart, python-tripleoclient in builds at [4]. Attaching a screenshot that shows the failed builds started 9th august.

[1] https://6e96b467b536ead9d981-83b5a515fd4f8ade122b2cc60973de6d.ssl.cf2.rackcdn.com/848640/2/check/tripleo-ci-centos-9-undercloud-upgrade/2a07467/logs/undercloud/home/zuul/undercloud_upgrade.log
[2] https://1eed5b8dc2d4d2c1a513-67e15cc69bd7500d8d48cf199d18d0c8.ssl.cf2.rackcdn.com/851958/4/gate/tripleo-ci-centos-9-undercloud-upgrade/8093ae0/logs/undercloud/home/zuul/undercloud_upgrade.log
[3] https://8cc16a860ef78508d8bf-63e0619ac9734be8bebe39a119169a2b.ssl.cf5.rackcdn.com/836979/2/gate/tripleo-ci-centos-9-undercloud-upgrade/fbfa910/logs/undercloud/home/zuul/undercloud_upgrade.log
[4] https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-9-undercloud-upgrade&skip=0

Revision history for this message
Marios Andreou (marios-b) wrote :
Revision history for this message
Marios Andreou (marios-b) wrote :

noting that a similar bug was filed a few months ago there https://bugs.launchpad.net/tripleo/+bug/1975623 root cause was mirror/sync issue

Revision history for this message
Marios Andreou (marios-b) wrote :

at http://mirror.stream.centos.org/9-stream/AppStream/x86_64/os/Packages/ I see

[ ] NetworkManager-ovs-1.39.5-1.el9.x86_64.rpm 2022-05-22 19:16 53K
[ ] NetworkManager-ovs-1.39.6-1.el9.x86_64.rpm 2022-06-07 07:46 53K
[ ] NetworkManager-ovs-1.39.7-2.el9.x86_64.rpm 2022-06-29 10:16 53K
[ ] NetworkManager-ovs-1.39.10-1.el9.x86_64.rpm 2022-07-19 09:16 54K
[ ] NetworkManager-ovs-1.39.12-1.el9.x86_64.rpm 2022-08-03 10:57 54K

So the 1.39.12-1 was updated over a week ago why is it complaining about it now?

Revision history for this message
Marios Andreou (marios-b) wrote :

        * http://mirror.stream.centos.org/9-stream/COMPOSE_ID

        * CentOS-Stream-9-20220808.1

That was 08th. We started seeing this on the 9th, might be related if this is mirror sync issue again

Revision history for this message
Marios Andreou (marios-b) wrote :

On the failing jobs the nodes have this networkmanager installed:

NetworkManager.x86_64 1:1.39.12-1.el9 @baseos
NetworkManager-libnm.x86_64 1:1.39.12-1.el9 @baseos

https://6e96b467b536ead9d981-83b5a515fd4f8ade122b2cc60973de6d.ssl.cf2.rackcdn.com/848640/2/check/tripleo-ci-centos-9-undercloud-upgrade/2a07467/logs/undercloud/var/log/extra/package-list-installed.txt

Revision history for this message
Marios Andreou (marios-b) wrote :

sent a note to list about rechecks @ https://lists.openstack.org/pipermail/openstack-discuss/2022-August/029942.html [tripleo] gate blocker tripleo-ci-centos-9-undercloud-upgrade

Revision history for this message
Alfredo Moralejo (amoralej) wrote :

Looks like AFS is out of sync.

Standard CentOS repo:

[root@77b891bf9fd8 ~]# dnf repoquery -q NetworkManager-ovs
NetworkManager-ovs-1:1.39.10-1.el9.x86_64
NetworkManager-ovs-1:1.39.12-1.el9.x86_64
NetworkManager-ovs-1:1.39.5-1.el9.x86_64
NetworkManager-ovs-1:1.39.6-1.el9.x86_64
NetworkManager-ovs-1:1.39.7-2.el9.x8

Checking AFS:

# dnf repoquery --repofrompath=afs,http://mirror.regionone.vexxhost-nodepool-sf.rdoproject.org/centos-stream/9-stream/AppStream/x86_64/os/ --disablerepo="*" --enablerepo=afs -q NetworkManager-ovs

NetworkManager-ovs-1:1.39.10-1.el9.x86_64
NetworkManager-ovs-1:1.39.3-1.el9.x86_64
NetworkManager-ovs-1:1.39.5-1.el9.x86_64
NetworkManager-ovs-1:1.39.6-1.el9.x86_64
NetworkManager-ovs-1:1.39.7-2.el9.x86_64

Revision history for this message
Jeremy Stanley (fungi) wrote :

  wget -qO- http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/ | grep NetworkManager-ovs | sed 's/.*href="\([^"]*\)".*/\1/'

  NetworkManager-ovs-1.39.3-1.el9.x86_64.rpm
  NetworkManager-ovs-1.39.5-1.el9.x86_64.rpm
  NetworkManager-ovs-1.39.6-1.el9.x86_64.rpm
  NetworkManager-ovs-1.39.7-2.el9.x86_64.rpm
  NetworkManager-ovs-1.39.10-1.el9.x86_64.rpm

Revision history for this message
Marios Andreou (marios-b) wrote :

adding for context as it isn't mentioned here only on the mailing list

amoralej posted https://review.opendev.org/c/opendev/system-config/+/852793 Use rackspace mirror to sync centos stream repos

the issue is ongoing at time of writing and the mirror has not yet synced.

https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-9-undercloud-upgrade&skip=0

if we merge 852793 it should unblock us

Revision history for this message
Marios Andreou (marios-b) wrote :

another similar bug there different packages but same repo and root cause https://bugs.launchpad.net/bugs/1984453

Revision history for this message
chandan kumar (chkumar246) wrote :

CentOS Stream 9 build infra was down on Aug 11 (https://lists.centos.org/pipermail/centos-devel/2022-August/120524.html). It might have impacted the facebook mirror, leading to out of sync for AFS mirror.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-ci (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/852880

Revision history for this message
Marios Andreou (marios-b) wrote (last edit ):

posted https://review.opendev.org/c/openstack/tripleo-ci/+/852880 to unblock gates if needed
even though i don't want to do that as we risk breaking that job or letting something through that should not be...

editing to add that i proposed the non-voting as this issue is blocking gates for 24 hours now

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-ci (master)

Change abandoned by "Ronelle Landy <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/852880

Revision history for this message
Sandeep Yadav (sandeepyadav93) wrote :

As per [1], Issue on Centos stream 9 build infra is solved

[1] https://lists.centos.org/pipermail/centos-devel/2022-August/120525.html

I see NetworkManager-ovs-1.39.12-1.el9.x86_64.rpm on FB mirror, rechecked the job to confirm if the issue is resolved.

~~~
$ wget -qO- http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/ | grep NetworkManager-ovs | sed 's/.*href="\([^"]*\)".*/\1/'

NetworkManager-ovs-1.39.5-1.el9.x86_64.rpm
NetworkManager-ovs-1.39.6-1.el9.x86_64.rpm
NetworkManager-ovs-1.39.7-2.el9.x86_64.rpm
NetworkManager-ovs-1.39.10-1.el9.x86_64.rpm
NetworkManager-ovs-1.39.12-1.el9.x86_64.rpm
~~~

Revision history for this message
chandan kumar (chkumar246) wrote :

Thank you Sandeep for checking it, the job https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-9-undercloud-upgrade&skip=0

https://zuul.openstack.org/build/a8d0b2716fbb45aeba63a8dd0a797584
tripleo-ci-centos-9-undercloud-upgrade SUCCESS is back to green now. I think we can mark this bug to resolved.

Changed in tripleo:
status: Triaged → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by "Marios Andreou <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/852880
Reason: luckily no longer needed - the bug is resolved/mirrors are OK again

Revision history for this message
Marios Andreou (marios-b) wrote (last edit ):

the change at https://review.opendev.org/c/opendev/system-config/+/852793 merged so I think that is actually what fixed us in this case.

The times seem to match up looking at the green runs in build history (attaching screenshot here).

[EDIT]: perhaps both things... per https://lists.centos.org/pipermail/centos-devel/2022-August/120525.html root cause of facebook mirror issue was fixed. and we also switched to rax mirror which seemed to be fine throughout.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.