Master/ Zed/Wallaby container build jobs are failing with error "Could not resolve host: mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org"

Centos 8 container build jobs are passing and only Centos 9 jobs are affected.

Centos 8 wallaby container build job history:

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-build-containers-centos-8-quay-push-wallaby

Centos 8 train container build job history:

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-build-containers-centos-8-quay-push-train

We see the same issue in Upstream infra:

https://c6e907431413b276a63b-23a7172f5dc1e58445390ec61883d218.ssl.cf2.rackcdn.com/864994/8/gate/tripleo-ci-centos-9-content-provider/9d99392/logs/container-builds/522fec05-8e3f-4cc7-8901-45f0ca049eed/base/os/horizon/horizon-build.log

~~~
Installed size: 6.8 M
Downloading Packages:
[MIRROR] python3-PyMySQL-0.10.1-6.el9.noarch.rpm: Curl error (6): Couldn't resolve host name for http://mirror.bhs1.ovh.opendev.org/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-PyMySQL-0.10.1-6.el9.noarch.rpm [Could not resolve host: mirror.bhs1.ovh.opendev.org]
[MIRROR] sscg-3.0.0-5.el9.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.bhs1.ovh.opendev.org/centos-stream/9-stream/AppStream/x86_64/os/Packages/sscg-3.0.0-5.el9.x86_64.rpm [Could not resolve host: mirror.bhs1.ovh.opendev.org]
[MIRROR] mod_ssl-2.4.53-7.el9.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.bhs1.ovh.opendev.org/centos-stream/9-stream/AppStream/x86_64/os/Packages/mod_ssl-2.4.53-7.el9.x86_64.rpm [Could not resolve host: mirror.bhs1.ovh.opendev.org]
[MIRROR] sscg-3.0.0-5.el9.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.bhs1.ovh.opendev.org/centos-stream/9-stream/AppStream/x86_64/os/Packages/sscg-3.0.0-5.el9.x86_64.rpm [Could not resolve host: mirror.bhs1.ovh.opendev.org]
[FAILED] sscg-3.0.0-5.el9.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success

The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Error downloading packages:
  sscg-3.0.0-5.el9.x86_64: Cannot download, all mirrors were already tried without success
~~~

Another example:

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_2ea/864469/1/gate/tripleo-ci-centos-9-content-provider/2ea04fe/logs/container-builds/3b86b709-7cd1-4929-bbc8-7ce5c35137a4/base/os/glance-api/glance-api-build.log

So it seems there are two patterns of failure.

1. In ovh, resolved.conf has the dns server set but DNS name resolution fails

2. In rax or rackcdn resolve.conf has no dns servers set.
 https://zuul.opendev.org/t/openstack/build/22500697c5244d31b0687057040cf1af

Though the both patterns are occurring not consistently but only occasionally.
We probably want to monitor the status before we determine whether the problem is a clear blocker or not.

Don't confuse where we store job builds logs with where jobs execute. We use OVH and rax/rackcdn based swift storage for storage of build logs. Jobs running in rackspace may upload to OVH swift and vice versa. All this to say that the location you see in the log url should not have any impact on job behavior that may be build provider specific. You need to check the build node names in the logs directly to determine where they ran.

Noted this on IRC but recording it here as well. At the beginning of the job we record the ansible host info. This host info shows that 127.0.0.1 is properly set as a DNS resolver: https://zuul.opendev.org/t/openstack/build/22500697c5244d31b0687057040cf1af/log/zuul-info/host-info.primary.yaml#187-189. But then after the jobs has failed an the job records logs it grabs the resolv.conf file which shows no resolvers are set: https://zuul.opendev.org/t/openstack/build/22500697c5244d31b0687057040cf1af/log/logs/undercloud/etc/resolv.conf.

This points to something updating the DNS configuration on the node while the job is running and when it does so appears to break DNS. For example in the links above we see no resolver is set, possibly because NetworkManager is attempting to set that info via DHCP but rackspace nodes do not do DHCP. Elsewhere DHCP may be overriding DNS resolvers to use local cloud resolvers which are overwhelmed by the number of requests? In any case this appears to be something in the job itself modifying the commit the node starts with when the job begins.

I have proposed a test patch to configure NetworkManager to not overwrite resolv.conf, /etc/resolv.conf is not overwritten after that during the job run.[2]

But we face the same DNS issue during the container image build.[3]

[1] https://review.opendev.org/c/openstack/tripleo-ci/+/865290/6/playbooks/tripleo-ci/test.yml#16

[2] https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e4d/865290/6/check/tripleo-ci-centos-9-content-provider/e4d0b10/logs/undercloud/etc/resolv.conf

~~~
nameserver 127.0.0.1
~~~

[3] https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e4d/865290/6/check/tripleo-ci-centos-9-content-provider/e4d0b10/logs/container-builds/0b349a4b-84f3-48f5-bd5a-b02eca881b57/base/os/horizon/horizon-build.log

~~~
[MIRROR] openstack-dashboard-23.1.0-0.20221107095346.c319118.el9.noarch.rpm: Curl error (28): Timeout was reached for http://mirror.mtl01.iweb.opendev.org:8080/rdo/centos9-master/component/ui/d8/5b/d85b61155590d61edbdc326dcd0e070c15f86fcb_6b89ae7d/openstack-dashboard-23.1.0-0.20221107095346.c319118.el9.noarch.rpm [Resolving timed out after 30000 milliseconds]
[FAILED] python-oslo-concurrency-lang-5.0.1-0.20220913081342.01cf2ff.el9.noarch.rpm: No more mirrors to try - All mirrors were already tried without success

The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Error downloading packages:
  python-oslo-concurrency-lang-5.0.1-0.20220913081342.01cf2ff.el9.noarch: Cannot download, all mirrors were already tried without success
~~~

Adding one observation:

Since Friday 18th Nov,

Container build in periodic promotion pipeline running on Vexxhost cloud is taking longer than usual.

Earlier those jobs used to finish within ~40 min and now they are taking longer than 1 hour.[1]

I noticed podman/buildah/netavark and some other rpm are bumped since Friday. Attaching diff of rpm.[2]

[1] https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-build-containers-ubi-9-quay-push-master&result=SUCCESS&skip=0
[2] https://www.diffchecker.com/9gfEnYH1/

Hello,

We tried excluding the latest netavark-1.3.*, aardvark-dns-1.3.*(as its a dep) and container build is back to its original time of ~40 mins (Without the exclude it was taking ~20 mins more)

https://review.rdoproject.org/zuul/build/6f230ae82e8a4f60aa7e38472de27572

Found a recent netavark issue[1] which may be affecting our jobs, Fix of this issue is not in netavark-1.3.*.

[1] https://github.com/containers/netavark/issues/491

Thanks to Rabi for patch[1] that switches to the host network for building containers(We already use the host network for running containers.)

Master patch merged and Zed cherry-pick is up.

[1] https://review.opendev.org/c/openstack/tripleo-common/+/865116
[2] https://review.opendev.org/c/openstack/tripleo-common/+/865391

No new failure for Content Provider on master/zed(job history[0]) since patches [1] and [2] merged

We are just waiting for wallaby's cherry-pick to merge[3].

[0] https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-9-content-provider&result=FAILURE&skip=0
[1] https://review.opendev.org/c/openstack/tripleo-common/+/865116
[2] https://review.opendev.org/c/openstack/tripleo-common/+/865391
[3] https://review.opendev.org/c/openstack/tripleo-common/+/865397