container quay-push jobs are failing push quay.rdoproject.org with 502 Bad Gateway

strangely this did not affect train, at [1] green runs 03 04 05 December

[1] https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-build-containers-centos-8-quay-push-train

OK, this looks like it is resolving? will monitor today and close the bug if so.

Zed [1] failed on 04th with this bug [2] but green today 05th

        * 2022-12-05 03:14:23 SUCCESS
        * 2022-12-04 13:22:07 FAILURE

[1] https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-build-containers-ubi-9-quay-push-zed
[2] https://logserver.rdoproject.org/openstack-periodic-integration-zed-centos9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-9-quay-push-zed/9fd5adc/logs/build.log

11:25 < dpawlik> marios|ruck: quay was out of disk space due
11:26 < dpawlik> zed repository was not added into the pruner
11:26 < dpawlik> that's why was 502

https://softwarefactory-project.io/r/c/software-factory/sf-infra/+/26900 Add api-centos9-zed into Quay pruner DLRN endpoints

Our Quay registry have an issue described here: https://issues.redhat.com/browse/PROJQUAY-2451 that has been fixed in https://github.com/quay/quay/pull/961 , but it requires upgrade to 3.7.0.

Update to 3.7.10 done, but space is cleaning up very slow.
The error 502 might happen once again.

this is still hitting us

example at [1] killed the c9/wallaby run this morning and [2] from the train line:

        * 2022-12-06 04:48:05 | 2022-12-06 04:48:05.119 39306 ERROR tripleoclient.v2.tripleo_container_image.Build Stderr: 'Getting image source signatures\nCopying blob sha256:ce9d08ae8f9927d284c3e9cd8e01fb1ccb35f3ccb58316e99172a48fb539f003\nCopying blob sha256:96b52c5f75bb0a72743ceab116e64ec82732f4b832df631ad97da614fb5ab311\nCopying blob sha256:ae04de67f65bc7e47c74b6897f995ceaaecfe3cd8bdd373f9f5409dd7d4e3b53\nCopying blob sha256:4e5109037c82ecb446faa91efeea0b88fd48579ea872a7df1a28a151f63a6814\nError: pushing image "quay.rdoproject.org/tripleowallabycentos9/openstack-ironic-base:b6edcbfef9c90c6251a7e5d45568f1ac" to "docker://quay.rdoproject.org/tripleowallabycentos9/openstack-ironic-base:b6edcbfef9c90c6251a7e5d45568f1ac": writing blob: initiating layer upload to /v2/tripleowallabycentos9/openstack-ironic-base/blobs/uploads/ in quay.rdoproject.org: received unexpected HTTP status: 502 Bad Gateway\n'

[1] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-9-quay-push-wallaby/50728b1/logs/build.log
[2] https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-centos-8-quay-push-train/7cffaa1/logs/build.log

moving this fix-released

dpawlik added a new volume yesterday
issue is resolved