tripleo

os_tempest: python_venv_build failing with "Depsolve Error occured: \n Problem: cannot install both gcc-8.5.0-7.el8.x86_64 and gcc-8.5.0-8.el8.x86_64\n

Bug #1959524 reported by Ronelle Landy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Unassigned
tripleo yoga-2

Bug Description

Tests running os_tempest are failing at the following step:

2022-01-30 06:09:51.120455 | primary | TASK [python_venv_build : Install distro packages for venv build] **************
2022-01-30 06:09:51.120561 | primary | Sunday 30 January 2022 06:09:51 +0000 (0:00:00.061) 0:40:46.553 ********
2022-01-30 06:09:54.199270 | primary | FAILED - RETRYING: Install distro packages for venv build (5 retries left).
2022-01-30 06:09:59.408360 | primary | FAILED - RETRYING: Install distro packages for venv build (4 retries left).
2022-01-30 06:10:04.410529 | primary | FAILED - RETRYING: Install distro packages for venv build (3 retries left).
2022-01-30 06:10:09.404302 | primary | FAILED - RETRYING: Install distro packages for venv build (2 retries left).
2022-01-30 06:10:14.448819 | primary | FAILED - RETRYING: Install distro packages for venv build (1 retries left).
2022-01-30 06:10:19.504967 | primary | fatal: [undercloud]: FAILED! => {"attempts": 5, "changed": false, "failures": [], "msg": "Depsolve Error occured: \n Problem: cannot install both gcc-8.5.0-7.el8.x86_64 and gcc-8.5.0-8.el8.x86_64\n - package gcc-c++-8.5.0-7.el8.x86_64 requires gcc = 8.5.0-7.el8, but none of the providers can be installed\n - cannot install the best update candidate for package gcc-8.5.0-8.el8.x86_64\n - cannot install the best candidate for the job", "rc": 1, "results": []}

https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-victoria/e9a13df/job-output.txt

Ronelle Landy (rlandy)
Changed in tripleo:
milestone: none → yoga-2
importance: Undecided → Critical
status: New → Triaged
tags: added: promotion-blocker
Revision history for this message
Ronelle Landy (rlandy) wrote :

This is hitting all tests:

https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario004-standalone-tripleo-master/baa75b9/job-output.txt

2022-01-30 20:27:44.192978 | primary | fatal: [undercloud]: FAILED! => {"attempts": 5, "changed": false, "failures": [], "msg": "Depsolve Error occured: \n Problem: cannot install both gcc-8.5.0-7.el8.x86_64 and gcc-8.5.0-8.el8.x86_64\n - package gcc-c++-8.5.0-7.el8.x86_64 requires gcc = 8.5.0-7.el8, but none of the providers can be installed\n - cannot install the best update candidate for package gcc-8.5.0-8.el8.x86_64\n - cannot install the best candidate for the job", "rc": 1, "results": []}

Error started on 01/28:
https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario004-standalone-tripleo-master

Revision history for this message
Ronelle Landy (rlandy) wrote :

gcc.x86_64 8.5.0-8.el8 @appstream

possible update from appstream?

Revision history for this message
Marios Andreou (marios-b) wrote (last edit ):

The error comes from [1] - a recent commit there is [2] but i don't know yet if that is responsible
as it merged 2 weeks ago... may still be a repo issue

[1] https://opendev.org/openstack/ansible-role-python_venv_build/src/commit/4a1490377ccde785d4f066f94608d18978894e0c/tasks/python_venv_install.yml#L32
[2] https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/824180

Revision history for this message
Marios Andreou (marios-b) wrote :

this is hitting all 8 branches e.g. master from today there [1]

2022-01-31 07:43:22.200095 | primary | fatal: [undercloud]: FAILED! => {"attempts": 5, "changed": false, "failures": [], "msg": "Depsolve Error occured: \n Problem: cannot install both gcc-8.5.0-7.el8.x86_64 and gcc-8.5.0-8.el8.x86_64\n - package gcc-c++-8.5.0-7.el8.x86_64 requires gcc = 8.5.0-7.el8, but none of the providers can be installed\n - cannot install the best update candidate for package gcc-8.5.0-8.el8.x86_64\n - cannot install the best candidate for the job", "rc": 1, "results": []}

[1] https://logserver.rdoproject.org/openstack-periodic-integration-main-centos8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-master/204a336/job-output.txt

Revision history for this message
Marios Andreou (marios-b) wrote :

http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/Packages/

        * gcc-c++-8.5.0-8.el8.x86_64.rpm 2022-01-27 16:09 12M
        * gcc-c++-8.5.0-7.el8.x86_64.rpm 2021-12-18 01:59 12M
        * gcc-8.5.0-7.el8.x86_64.rpm 2021-12-18 01:59 23M
        * gcc-8.5.0-8.el8.x86_64.rpm 2022-01-27 16:09 23M

Revision history for this message
Alfredo Moralejo (amoralej) wrote :

Synchronization to AFS happened properly on 28th January. Both gcc and gcc-c++ were updated together:

centos.log.1:2022-01-28T16:43:02,784921420+00:00
centos.log.1:AppStream/aarch64/os/Packages/gcc-8.5.0-8.el8.aarch64.rpm
centos.log.1:AppStream/aarch64/os/Packages/gcc-c++-8.5.0-8.el8.aarch64.rpm
centos.log.1:AppStream/aarch64/os/Packages/gcc-gdb-plugin-8.5.0-8.el8.aarch64.rpm
centos.log.1:AppStream/aarch64/os/Packages/gcc-gfortran-8.5.0-8.el8.aarch64.rpm
centos.log.1:AppStream/aarch64/os/Packages/libstdc++-devel-8.5.0-8.el8.aarch64.rpm
centos.log.1:AppStream/aarch64/os/Packages/libstdc++-docs-8.5.0-8.el8.aarch64.rpm
centos.log.1:AppStream/x86_64/os/Packages/gcc-8.5.0-8.el8.x86_64.rpm
centos.log.1:AppStream/x86_64/os/Packages/gcc-c++-8.5.0-8.el8.x86_64.rpm
centos.log.1:AppStream/x86_64/os/Packages/gcc-gdb-plugin-8.5.0-8.el8.i686.rpm
centos.log.1:AppStream/x86_64/os/Packages/gcc-gdb-plugin-8.5.0-8.el8.x86_64.rpm
centos.log.1:AppStream/x86_64/os/Packages/gcc-gfortran-8.5.0-8.el8.x86_64.rpm
centos.log.1:AppStream/x86_64/os/Packages/gcc-offload-nvptx-8.5.0-8.el8.x86_64.rpm
centos.log.1:AppStream/x86_64/os/Packages/libstdc++-devel-8.5.0-8.el8.i686.rpm
centos.log.1:AppStream/x86_64/os/Packages/libstdc++-devel-8.5.0-8.el8.x86_64.rpm
centos.log.1:AppStream/x86_64/os/Packages/libstdc++-docs-8.5.0-8.el8.x86_64.rpm
centos.log.1:BaseOS/aarch64/os/Packages/libgcc-8.5.0-8.el8.aarch64.rpm
centos.log.1:BaseOS/aarch64/os/Packages/libgfortran-8.5.0-8.el8.aarch64.rpm
centos.log.1:BaseOS/aarch64/os/Packages/libstdc++-8.5.0-8.el8.aarch64.rpm
centos.log.1:BaseOS/x86_64/os/Packages/libgcc-8.5.0-8.el8.i686.rpm
centos.log.1:BaseOS/x86_64/os/Packages/libgcc-8.5.0-8.el8.x86_64.rpm
centos.log.1:BaseOS/x86_64/os/Packages/libgfortran-8.5.0-8.el8.i686.rpm
centos.log.1:BaseOS/x86_64/os/Packages/libgfortran-8.5.0-8.el8.x86_64.rpm
centos.log.1:BaseOS/x86_64/os/Packages/libstdc++-8.5.0-8.el8.i686.rpm
centos.log.1:BaseOS/x86_64/os/Packages/libstdc++-8.5.0-8.el8.x86_64.rpm

Something i don't understan well is why in those jobs we have installed:

libgfortran-8.5.0-7.el8.x86_64 (note 7)
gcc-8.5.0-8.el8.x86_64

https://logserver.rdoproject.org/openstack-periodic-integration-main-centos8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-master/204a336/logs/undercloud/var/log/extra/rpm-list.txt.gz

still investigating

Revision history for this message
Marios Andreou (marios-b) wrote :

the openstack-periodic-integration-main-centos8 is currently running but it looks like the issue is resolved in today's run

we have green scen1/2/3/4/7/12

e.g.

https://logserver.rdoproject.org/openstack-periodic-integration-main-centos8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-master/0bebe01/

https://logserver.rdoproject.org/openstack-periodic-integration-main-centos8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-master/031ed9f/

https://logserver.rdoproject.org/openstack-periodic-integration-main-centos8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario003-standalone-master/91d3bab/

... so confirmed 'mirror issue' ? :/

Revision history for this message
Alfredo Moralejo (amoralej) wrote :

For the record, it seems we were only hitting this in jobs running in SF zuul:

<marios|ruck> amoralej: i don't think we saw that one in gates it was only in the rdo SF/periodic jobs

I couldn't find anything wrong in the repos sync.

I suspect that the issue may have been caused for something in the image, but i'm not sure what.

Revision history for this message
Marios Andreou (marios-b) wrote :

we are mostly clear of this but for some reason the train line failed today there

        * https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/051457b7cd20347240a871126d2943d20722b78c/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-train/19b7d46/logs/undercloud/home/zuul/build.log.txt.gz

        * 2022-02-02 03:16:45.060 | > Error:
          2022-02-02 03:16:45.060 | > Problem: cannot install the best candidate for the job
          2022-02-02 03:16:45.060 | > - nothing provides libgcc >= 8.5.0-10.el8 needed by gcc-8.5.0-10.el8.x86_64
          2022-02-02 03:16:45.060 | > - nothing provides libgomp = 8.5.0-10.el8 needed by gcc-8.5.0-10.el8.x86_64

Revision history for this message
Alfredo Moralejo (amoralej) wrote :

I've investigated this particular error which is different that the one reported originally. So the issue in this case (as we had previously) it's appstream and baseos repos being out of sync because they are picking different mirrors.

This shouldn't happen if we use AFS mirrors for the mirrors as they sync from the same source repo server, but in this case we are including both the repos config with AFS mirrors and the default ones using mirrorlist:

2022-02-02 02:51:13.289 | CentOS Stream 8 - AppStream 17 MB/s | 19 MB 00:01
2022-02-02 02:51:20.665 | CentOS Stream 8 - BaseOS 5.7 MB/s | 18 MB 00:03
2022-02-02 02:51:25.086 | CentOS Stream 8 - Extras 19 kB/s | 17 kB 00:00
2022-02-02 02:51:25.572 | dlrn-train-testing 5.8 MB/s | 1.5 MB 00:00
2022-02-02 02:51:26.369 | dlrn-train-build-deps 1.5 MB/s | 395 kB 00:00
2022-02-02 02:51:26.732 | Advanced Virtualization mirror 3.1 MB/s | 207 kB 00:00
2022-02-02 02:51:27.048 | Messaging RabbitMQ 1.3 MB/s | 80 kB 00:00
2022-02-02 02:51:27.348 | opstools 943 kB/s | 149 kB 00:00
2022-02-02 02:51:27.732 | centos-8-fix 119 kB/s | 19 kB 00:00
2022-02-02 02:51:28.286 | quickstart-centos-appstreams 55 MB/s | 19 MB 00:00
2022-02-02 02:51:33.005 | quickstart-centos-base 62 MB/s | 18 MB 00:00
2022-02-02 02:51:36.977 | quickstart-centos-ceph-nautilus 3.0 MB/s | 549 kB 00:00
2022-02-02 02:51:37.346 | quickstart-centos-extras 275 kB/s | 16 kB 00:00
2022-02-02 02:51:37.579 | quickstart-centos-highavailability 21 MB/s | 2.2 MB 00:00
2022-02-02 02:51:38.557 | quickstart-centos-opstools 418 kB/s | 149 kB 00:00
2022-02-02 02:51:38.837 | quickstart-centos-powertools 32 MB/s | 4.0 MB 00:00

2022-02-02 03:16:44.429 | > repo: using cache for: appstream
2022-02-02 03:16:44.429 | > appstream: using metadata from Tue Feb 1 18:18:39 2022.
2022-02-02 03:16:44.429 | > repo: using cache for: baseos
2022-02-02 03:16:44.429 | > baseos: using metadata from Fri Jan 28 17:04:05 2022.

2022-02-02 03:16:44.429 | > repo: using cache for: quickstart-centos-appstreams
2022-02-02 03:16:44.429 | > quickstart-centos-appstreams: using metadata from Fri Jan 28 17:05:04 2022.
2022-02-02 03:16:44.429 | > repo: using cache for: quickstart-centos-base
2022-02-02 03:16:44.430 | > quickstart-centos-base: using metadata from Fri Jan 28 17:04:05 2022.

CentOS should provide some way to avoid this kind of issues but there is not a solution from their side yet.

I'd say we should ensure we are only using the AFS ones for all centos repos, i think that's our best bet so far.

Ronelle Landy (rlandy)
Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.