tripleo

tripleo-ci-centos-8-content-provider failed to pull containers from quay.ceph.io

Bug #1927952 reported by Pooja Jadhav on 2021-05-10

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Fix Released	Critical	Unassigned	tripleo xena-3

Bug Description

tripleo-ci-centos-8-content-provider failed to pull contaimers from quay.ceph.io having below traceback :

2021-05-10 09:00:10.329814 | primary | TASK [container-build : List down missing containers] **************************
2021-05-10 09:00:10.329982 | primary | Monday 10 May 2021 09:00:10 +0000 (0:00:13.204) 0:55:32.166 ************
2021-05-10 09:00:10.438524 | primary | ok: [undercloud] => (item={'msg': 'non-zero return code', 'cmd': ['podman', 'pull', 'docker://trunk.registry.rdoproject.org/ceph/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64'], 'stdout': '', 'stderr': 'Trying to pull docker://trunk.registry.rdoproject.org/ceph/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64...\n manifest unknown: manifest unknown\nError: 1 error occurred:\n\t* Error initializing source docker://trunk.registry.rdoproject.org/ceph/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64: Error reading manifest v6.0.0-stable-6.0-pacific-centos-8-x86_64 in trunk.registry.rdoproject.org/ceph/daemon: manifest unknown: manifest unknown', 'rc': 125, 'start': '2021-05-10 08:59:57.597731', 'end': '2021-05-10 08:59:58.507648', 'delta': '0:00:00.909917', 'changed': True, 'failed': True, 'invocation': {'module_args': {'_raw_params': 'podman pull docker://trunk.registry.rdoproject.org/ceph/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64', 'warn': False, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [], 'stderr_lines': ['Trying to pull docker://trunk.registry.rdoproject.org/ceph/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64...', ' manifest unknown: manifest unknown', 'Error: 1 error occurred:', '\t* Error initializing source docker://trunk.registry.rdoproject.org/ceph/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64: Error reading manifest v6.0.0-stable-6.0-pacific-centos-8-x86_64 in trunk.registry.rdoproject.org/ceph/daemon: manifest unknown: manifest unknown'], 'item': 'quay.ceph.io/ceph-ci/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64', 'ansible_loop_var': 'item'})
2021-05-10 09:00:10.487931 | primary | skipping: [undercloud] => (item={'cmd': ['podman', 'pull', 'docker://trunk.registry.rdoproject.org/ceph/prometheus:v2.7.2'], 'stdout': '2a68492b2466243c7c73692f575e04c091feebfeae94bb164c4cc811bed773b0', 'stderr': 'Trying to pull docker://trunk.registry.rdoproject.org/ceph/prometheus:v2.7.2...\nGetting image source signatures\nCopying blob sha256:75c520aff8e8d376ed1b5d18fd8b8df5bd2e4a02a0154ceab9af9d5dfdea5f01\nCopying blob sha256:6d73410cc7fa96bdea537862c4d160d29d252949b71ab67f000d0e7a7c27022c\nCopying blob sha256:494d1a723657236b2a19ba8773d2058afc39579f2d5f760d9841e22746d84461\nCopying blob sha256:57c14dd66db0390dbf6da578421c077f6de8e88edd0815b4caa94607ba5f4c09\nCopying blob sha256:5f8f01ea2931bc86dba46415aca1f80f73c1ee67f18d34f4376ca1d365811bbd\nCopying blob sha256:65b1a901dcccc7c92e3b43591444a08b0b6ec104758a301c04a02dfb05500c67\nCopying blob sha256:c0fd0ac7829e8ad3f4d3e1a6bab933b2253440566b121e2cd2d5a790c9d3d09b\nCopying blob sha256:89c95bc388e90cb3fceb3a44f5f852533602db9ff54e9795e50326cfd96344a4\nCopying blob sha256:e51130820cb7f52e561a2f05abd624b0793bcc36accf3b2e5f5d2abdcb4fe6f2\nCopying config sha256:2a68492b2466243c7c73692f575e04c091feebfeae94bb164c4cc811bed773b0\nWriting manifest to image destination\nStoring signatures', 'rc': 0, 'start': '2021-05-10 08:59:58.969999', 'end': '2021-05-10 09:00:03.444637', 'delta': '0:00:04.474638', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'podman pull docker://trunk.registry.rdoproject.org/ceph/prometheus:v2.7.2', 'warn': False, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['2a68492b2466243c7c73692f575e04c091feebfeae94bb164c4cc811bed773b0'], 'stderr_lines': ['Trying to pull docker://trunk.registry.rdoproject.org/ceph/prometheus:v2.7.2...', 'Getting image source signatures', 'Copying blob sha256:75c520aff8e8d376ed1b5d18fd8b8df5bd2e4a02a0154ceab9af9d5dfdea5f01', 'Copying blob sha256:6d73410cc7fa96bdea537862c4d160d29d252949b71ab67f000d0e7a7c27022c', 'Copying blob sha256:494d1a723657236b2a19ba8773d2058afc39579f2d5f760d9841e22746d84461', 'Copying blob sha256:57c14dd66db0390dbf6da578421c077f6de8e88edd0815b4caa94607ba5f4c09', 'Copying blob sha256:5f8f01ea2931bc86dba46415aca1f80f73c1ee67f18d34f4376ca1d365811bbd', 'Copying blob sha256:65b1a901dcccc7c92e3b43591444a08b0b6ec104758a301c04a02dfb05500c67', 'Copying blob sha256:c0fd0ac7829e8ad3f4d3e1a6bab933b2253440566b121e2cd2d5a790c9d3d09b', 'Copying blob sha256:89c95bc388e90cb3fceb3a44f5f852533602db9ff54e9795e50326cfd96344a4', 'Copying blob sha256:e51130820cb7f52e561a2f05abd624b0793bcc36accf3b2e5f5d2abdcb4fe6f2', 'Copying config sha256:2a68492b2466243c7c73692f575e04c091feebfeae94bb164c4cc811bed773b0', 'Writing manifest to image destination', 'Storing signatures'], 'failed': False, 'item': 'quay.ceph.io/prometheus/prometheus:v2.7.2', 'ansible_loop_var': 'item'})
2021-05-10 09:00:10.538197 | primary | skipping: [undercloud] => (item={'cmd': ['podman', 'pull', 'docker://trunk.registry.rdoproject.org/ceph/alertmanager:v0.16.2'], 'stdout': 'a1c1bd95a2dd2b45df13eb39a56e25c2f1b791d7ffde6fb1fe9cddbedc99bd01', 'stderr': 'Trying to pull docker://trunk.registry.rdoproject.org/ceph/alertmanager:v0.16.2...\nGetting image source signatures\nCopying blob sha256:4ee3d6d19c59903413e1e6b55b29e2dd2268831c41ac5489511b39c1a4347065\nCopying blob sha256:ed2709e725224b326016c74f92927541229642f5bdbb7a38980aa88d291fc075\nCopying blob sha256:697743189b6d255069caf6c455be10c7f8cae8076c6f94d224ae15cd41420e87\nCopying blob sha256:a94ecdcbbebe73bf02cee692ccc517fb51e9d08d6fc4dc09017fbf139b5cfba3\nCopying blob sha256:85bad536b567cf559bf80da1bab180dabfc8d96cdd3d69efaf6a61a786d7e229\nCopying blob sha256:9ed97210a3829d5624e829e1ede7c794568b670dd88942b873337fe5fd806b76\nCopying config sha256:a1c1bd95a2dd2b45df13eb39a56e25c2f1b791d7ffde6fb1fe9cddbedc99bd01\nWriting manifest to image destination\nStoring signatures', 'rc': 0, 'start': '2021-05-10 09:00:03.903793', 'end': '2021-05-10 09:00:06.849636', 'delta': '0:00:02.945843', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'podman pull docker://trunk.registry.rdoproject.org/ceph/alertmanager:v0.16.2', 'warn': False, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['a1c1bd95a2dd2b45df13eb39a56e25c2f1b791d7ffde6fb1fe9cddbedc99bd01'], 'stderr_lines': ['Trying to pull docker://trunk.registry.rdoproject.org/ceph/alertmanager:v0.16.2...', 'Getting image source signatures', 'Copying blob sha256:4ee3d6d19c59903413e1e6b55b29e2dd2268831c41ac5489511b39c1a4347065', 'Copying blob sha256:ed2709e725224b326016c74f92927541229642f5bdbb7a38980aa88d291fc075', 'Copying blob sha256:697743189b6d255069caf6c455be10c7f8cae8076c6f94d224ae15cd41420e87', 'Copying blob sha256:a94ecdcbbebe73bf02cee692ccc517fb51e9d08d6fc4dc09017fbf139b5cfba3', 'Copying blob sha256:85bad536b567cf559bf80da1bab180dabfc8d96cdd3d69efaf6a61a786d7e229', 'Copying blob sha256:9ed97210a3829d5624e829e1ede7c794568b670dd88942b873337fe5fd806b76', 'Copying config sha256:a1c1bd95a2dd2b45df13eb39a56e25c2f1b791d7ffde6fb1fe9cddbedc99bd01', 'Writing manifest to image destination', 'Storing signatures'], 'failed': False, 'item': 'quay.ceph.io/prometheus/alertmanager:v0.16.2', 'ansible_loop_var': 'item'})
2021-05-10 09:00:10.588579 | primary | skipping: [undercloud] => (item={'cmd': ['podman', 'pull', 'docker://trunk.registry.rdoproject.org/ceph/node-exporter:v0.17.0'], 'stdout': 'b3e7f67a1480979f7c2068c14e4f0d3da6243631b9def8490ed68236a8311f11', 'stderr': 'Trying to pull docker://trunk.registry.rdoproject.org/ceph/node-exporter:v0.17.0...\nGetting image source signatures\nCopying blob sha256:0de338cf42586fa81850a9f1610429364d1a7f348e8db87786be48663b25a60d\nCopying blob sha256:d764f7880123ec20906e97bf9830c1fdc0f741549d88b3145ae8ac8da0017b2e\nCopying blob sha256:f508012419d8ed023913de91bf25eef8192444e7624e71cb87c5ce6d7323d497\nCopying config sha256:b3e7f67a1480979f7c2068c14e4f0d3da6243631b9def8490ed68236a8311f11\nWriting manifest to image destination\nStoring signatures', 'rc': 0, 'start': '2021-05-10 09:00:07.296284', 'end': '2021-05-10 09:00:09.330567', 'delta': '0:00:02.034283', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'podman pull docker://trunk.registry.rdoproject.org/ceph/node-exporter:v0.17.0', 'warn': False, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['b3e7f67a1480979f7c2068c14e4f0d3da6243631b9def8490ed68236a8311f11'], 'stderr_lines': ['Trying to pull docker://trunk.registry.rdoproject.org/ceph/node-exporter:v0.17.0...', 'Getting image source signatures', 'Copying blob sha256:0de338cf42586fa81850a9f1610429364d1a7f348e8db87786be48663b25a60d', 'Copying blob sha256:d764f7880123ec20906e97bf9830c1fdc0f741549d88b3145ae8ac8da0017b2e', 'Copying blob sha256:f508012419d8ed023913de91bf25eef8192444e7624e71cb87c5ce6d7323d497', 'Copying config sha256:b3e7f67a1480979f7c2068c14e4f0d3da6243631b9def8490ed68236a8311f11', 'Writing manifest to image destination', 'Storing signatures'], 'failed': False, 'item': 'quay.ceph.io/prometheus/node-exporter:v0.17.0', 'ansible_loop_var': 'item'})
2021-05-10 09:00:10.650911 | primary | ok: [undercloud] => (item={'msg': 'non-zero return code', 'cmd': ['podman', 'pull', 'docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4'], 'stdout': '', 'stderr': 'Trying to pull docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4...\n manifest unknown: manifest unknown\nError: 1 error occurred:\n\t* Error initializing source docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4: Error reading manifest 6.7.4 in trunk.registry.rdoproject.org/ceph/grafana: manifest unknown: manifest unknown', 'rc': 125, 'start': '2021-05-10 09:00:09.787072', 'end': '2021-05-10 09:00:10.282573', 'delta': '0:00:00.495501', 'changed': True, 'failed': True, 'invocation': {'module_args': {'_raw_params': 'podman pull docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4', 'warn': False, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [], 'stderr_lines': ['Trying to pull docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4...', ' manifest unknown: manifest unknown', 'Error: 1 error occurred:', '\t* Error initializing source docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4: Error reading manifest 6.7.4 in trunk.registry.rdoproject.org/ceph/grafana: manifest unknown: manifest unknown'], 'item': 'quay.ceph.io/app-sre/grafana:6.7.4', 'ansible_loop_var': 'item'})

References :

https://33807c6af0e20453b14c-c0af318d07ef9bf7ba9a60c77add6086.ssl.cf5.rackcdn.com/790377/1/check/tripleo-ci-centos-8-content-provider/2e666e0/job-output.txt

Tags:

Rabi Mishra (rabi) on 2021-05-10

Changed in tripleo:
importance:	Undecided → Critical

wes hayutin (weshayutin) on 2021-05-10

tags:

added: promotion-blocker

Revision history for this message

wes hayutin (weshayutin) wrote on 2021-05-10:

Download full text (5.9 KiB)

ceph related containers have been pushed into rdo.. however we still have a problem.

ceph related containers have been pushed into rdo.. however we still have a problem.

2021-05-10 14:45:06.018167 | primary | 127.0.0.1:5001/tripleomaster/openstack-base                         22fd7520a99b928db16ffaa8eaee9b63            f3a164bc5eab   40 minutes ago   561 MB
2021-05-10 14:45:06.018193 | primary | registry.access.redhat.com/ubi8                                     latest                                      613e5da7a934   3 weeks ago      213 MB
2021-05-10 14:45:06.018221 | primary | trunk.registry.rdoproject.org/ceph/daemon                           v6.0.0-stable-6.0-pacific-centos-8-x86_64   14fee0875498   5 weeks ago      1.17 GB
2021-05-10 14:45:06.018249 | primary | quay.io/tripleoci/registry                                          2                                           2d4f4b5309b1   10 months ago    26.8 MB
2021-05-10 14:45:06.018265 | primary | trunk.registry.rdoproject.org/ceph/grafana                          6.7.4                                       be4c69a1aae8   11 months ago    237 MB
2021-05-10 14:45:06.018278 | primary | trunk.registry.rdoproject.org/ceph/alertmanager                     v0.16.2                                     a1c1bd95a2dd   2 years ago      43.9 MB
2021-05-10 14:45:06.018291 | primary | trunk.registry.rdoproject.org/ceph/prometheus                       v2.7.2                                      2a68492b2466   2 years ago      103 MB
2021-05-10 14:45:06.018303 | primary | trunk.registry.rdoproject.org/ceph/node-exporter                    v0.17.0                                     b3e7f67a1480   2 years ago      22.4 MB
2021-05-10 14:45:06.018315 | primary |
2021-05-10 14:45:06.018328 | primary |
2021-05-10 14:45:06.018341 | primary | STDERR:
2021-05-10 14:45:06.018353 | primary |
2021-05-10 14:45:06.018365 | primary | level=debug msg="[graphdriver] trying provided driver \"overlay\""
2021-05-10 14:45:06.018378 | primary | level=debug msg="cached value indicated that overlay is supported"
2021-05-10 14:45:06.018390 | primary | level=debug msg="cached value indicated that metacopy is being used"
2021-05-10 14:45:06.018402 | primary | level=debug msg="cached value indicated that native-diff is not being used"
2021-05-10 14:45:06.018415 | primary | level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
2021-05-10 14:45:06.018428 | primary | level=debug msg="backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true"
2021-05-10 14:45:06.018440 | primary | level=debug msg="Loading registries configuration \"/etc/containers/registries.conf\""
2021-05-10 14:45:06.018453 | primary | level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/000-shortnames.conf\""
2021-05-10 14:45:06.018466 | primary | level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/rhel-shortnames.conf\""
2021-05-10 14:45:06.018479 | primary | level=debug msg="registry \"127.0.0.1:5001\" is not listed in registries configuration \"/etc/containers/registries.conf\", assuming it's not blocked"
2021-05-10 14:45:06.018491 | primary | level=debug msg="parsed reference into \"[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]quay.ceph.io/prometheus/node-exporter:v0.17.0\""
2021-05-10 14:45:06.018505 | primary | level=debug msg="reference \"[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]quay.ceph.io/prometheus/node-exporter:v0.17.0\" does not resolve to an image ID"
2021-05-10 14:45:06.018525 | primary | level=debug msg="error locating image \"quay.ceph.io/prometheus/node-exporter:v0.17.0\": error locating image with ID \"quay.ceph.io/prometheus/node-exporter:v0.17.0\": image not known"
2021-05-10 14:45:06.018540 | primary | error pushing image "quay.ceph.io/prometheus/node-exporter:v0.17.0" to "docker://127.0.0.1:5001/tripleomaster/node-exporter:v0.17.0": error locating image with name "quay.ceph.io/prometheus/node-exporter:v0.17.0" ([quay.ceph.io/prometheus/node-exporter:v0.17.0]): image not known
2021-05-10 14:45:06.018554 | primary | level=debug msg="shutting down the store"
2021-05-10 14:45:06.018567 | primary | level=info msg="failed to shutdown storage: \"A layer is mounted: layer is in use by a container\""
2021-05-10 14:45:06.018579 | primary |
2021-05-10 14:45:06.018592 | primary |
2021-05-10 14:45:06.018604 | primary | MSG:
2021-05-10 14:45:06.018617 | primary |
2021-05-10 14:45:06.018629 | primary | non-zero return code
2021-05-10 14:45:06.907049 | primary | FAILED - RETRYING: Pull non-tripleo containers (ceph, alertmanager, prometheus) to the content provider registry (3 retries left).
2021-05-10 14:45:12.749587 | primary | FAILED - RETRYING: Pull non-tripleo containers (ceph, alertmanager, prometheus) to the content provider registry (2 retries left).
2021-05-10 14:45:18.624907 | primary | FAILED - RETRYING: Pull non-tripleo containers (ceph, alertmanager, prometheus) to the content provider registry (1 retries left).
2021-05-10 14:45:24.507723 | primary | failed: [undercloud] (item=quay.ceph.io/app-sre/grafana:6.7.4) => {
2021-05-10 14:45:24.507786 | primary |     "ansible_loop_var": "item",
2021-05-10 14:45:24.507805 | primary |     "attempts": 3,
2021-05-10 14:45:24.507819 | primary |     "changed": false,
2021-05-10 14:45:24.507833 | primary |     "cmd": "buildah images; buildah push --format=v2s2 --tls-verify=False --log-level debug quay.ceph.io/app-sre/grafana:6.7.4  docker://127.0.0.1:5001/tripleomaster/grafana:6.7.4;\n",
2021-05-10 14:45:24.507847 | primary |     "delta": "0:00:00.462100",
2021-05-10 14:45:24.507860 | primary |     "end": "2021-05-10 14:45:24.466558",
2021-05-10 14:45:24.507907 | primary |     "item": "quay.ceph.io/app-sre/grafana:6.7.4",
2021-05-10 14:45:24.507946 | primary |     "rc": 125,
2021-05-10 14:45:24.507969 | primary |     "start": "2021-05-10 14:45:24.004458"
2021-05-10 14:45:24.507983 | primary | }

Revision history for this message

wes hayutin (weshayutin) wrote on 2021-05-10:

https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/782362 in play