I've investigated this particular error which is different that the one reported originally. So the issue in this case (as we had previously) it's appstream and baseos repos being out of sync because they are picking different mirrors.
This shouldn't happen if we use AFS mirrors for the mirrors as they sync from the same source repo server, but in this case we are including both the repos config with AFS mirrors and the default ones using mirrorlist:
I've investigated this particular error which is different that the one reported originally. So the issue in this case (as we had previously) it's appstream and baseos repos being out of sync because they are picking different mirrors.
This shouldn't happen if we use AFS mirrors for the mirrors as they sync from the same source repo server, but in this case we are including both the repos config with AFS mirrors and the default ones using mirrorlist:
2022-02-02 02:51:13.289 | CentOS Stream 8 - AppStream 17 MB/s | 19 MB 00:01 build-deps 1.5 MB/s | 395 kB 00:00 centos- appstreams 55 MB/s | 19 MB 00:00 centos- base 62 MB/s | 18 MB 00:00 centos- ceph-nautilus 3.0 MB/s | 549 kB 00:00 centos- extras 275 kB/s | 16 kB 00:00 centos- highavailabilit y 21 MB/s | 2.2 MB 00:00 centos- opstools 418 kB/s | 149 kB 00:00 centos- powertools 32 MB/s | 4.0 MB 00:00
2022-02-02 02:51:20.665 | CentOS Stream 8 - BaseOS 5.7 MB/s | 18 MB 00:03
2022-02-02 02:51:25.086 | CentOS Stream 8 - Extras 19 kB/s | 17 kB 00:00
2022-02-02 02:51:25.572 | dlrn-train-testing 5.8 MB/s | 1.5 MB 00:00
2022-02-02 02:51:26.369 | dlrn-train-
2022-02-02 02:51:26.732 | Advanced Virtualization mirror 3.1 MB/s | 207 kB 00:00
2022-02-02 02:51:27.048 | Messaging RabbitMQ 1.3 MB/s | 80 kB 00:00
2022-02-02 02:51:27.348 | opstools 943 kB/s | 149 kB 00:00
2022-02-02 02:51:27.732 | centos-8-fix 119 kB/s | 19 kB 00:00
2022-02-02 02:51:28.286 | quickstart-
2022-02-02 02:51:33.005 | quickstart-
2022-02-02 02:51:36.977 | quickstart-
2022-02-02 02:51:37.346 | quickstart-
2022-02-02 02:51:37.579 | quickstart-
2022-02-02 02:51:38.557 | quickstart-
2022-02-02 02:51:38.837 | quickstart-
2022-02-02 03:16:44.429 | > repo: using cache for: appstream
2022-02-02 03:16:44.429 | > appstream: using metadata from Tue Feb 1 18:18:39 2022.
2022-02-02 03:16:44.429 | > repo: using cache for: baseos
2022-02-02 03:16:44.429 | > baseos: using metadata from Fri Jan 28 17:04:05 2022.
2022-02-02 03:16:44.429 | > repo: using cache for: quickstart- centos- appstreams centos- appstreams: using metadata from Fri Jan 28 17:05:04 2022. centos- base centos- base: using metadata from Fri Jan 28 17:04:05 2022.
2022-02-02 03:16:44.429 | > quickstart-
2022-02-02 03:16:44.429 | > repo: using cache for: quickstart-
2022-02-02 03:16:44.430 | > quickstart-
CentOS should provide some way to avoid this kind of issues but there is not a solution from their side yet.
I'd say we should ensure we are only using the AFS ones for all centos repos, i think that's our best bet so far.