Most of the content provider jobs are failing as quay.io/app-sre/grafana:6.7.4 image does not exists on quay.io.
Here is the traceback:
https://447f476af5555fa473a2-ba0bbef8fa5bd9d33ddbd8694210833c.ssl.cf5.rackcdn.com/781622/2/check/tripleo-ci-centos-8-content-provider/e902500/logs/quickstart_install.log
```
TASK [container-build : Pull failed containers from RDO registry] **************
Tuesday 23 March 2021 05:35:37 +0000 (0:00:00.230) 0:46:04.613 *********
failed: [undercloud] (item=quay.io/app-sre/grafana:6.7.4) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": [
"podman",
"pull",
"docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4"
],
"delta": "0:00:00.874778",
"end": "2021-03-23 05:35:38.377970",
"item": "quay.io/app-sre/grafana:6.7.4",
"rc": 125,
"start": "2021-03-23 05:35:37.503192"
}
STDERR:
Trying to pull docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4...
manifest unknown: manifest unknown
Error: error pulling image "docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4": unable to pull docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4: unable to pull image: Error initializing source docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4: Error reading manifest 6.7.4 in trunk.registry.rdoproject.org/ceph/grafana: manifest unknown: manifest unknown
MSG:
non-zero return code
...ignoring
TASK [container-build : List down missing containers] **************************
Tuesday 23 March 2021 05:35:38 +0000 (0:00:01.337) 0:46:05.951 *********
ok: [undercloud] => (item={'msg': 'non-zero return code', 'cmd': ['podman', 'pull', 'docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4'], 'stdout': '', 'stderr': 'Trying to pull docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4...\n manifest unknown: manifest unknown\nError: error pulling image "docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4": unable to pull docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4: unable to pull image: Error initializing source docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4: Error reading manifest 6.7.4 in trunk.registry.rdoproject.org/ceph/grafana: manifest unknown: manifest unknown', 'rc': 125, 'start': '2021-03-23 05:35:37.503192', 'end': '2021-03-23 05:35:38.377970', 'delta': '0:00:00.874778', 'changed': True, 'failed': True, 'invocation': {'module_args': {'_raw_params': 'podman pull docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4', 'warn': False, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [], 'stderr_lines': ['Trying to pull docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4...', ' manifest unknown: manifest unknown', 'Error: error pulling image "docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4": unable to pull docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4: unable to pull image: Error initializing source docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4: Error reading manifest 6.7.4 in trunk.registry.rdoproject.org/ceph/grafana: manifest unknown: manifest unknown'], 'item': 'quay.io/app-sre/grafana:6.7.4', 'ansible_loop_var': 'item'})
TASK [container-build : Pull failed containers from dockerhub] *****************
Tuesday 23 March 2021 05:35:38 +0000 (0:00:00.117) 0:46:06.069 *********
failed: [undercloud] (item=quay.io/app-sre/grafana:6.7.4) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": [
"podman",
"pull",
"grafana:6.7.4"
],
"delta": "0:00:00.716062",
"end": "2021-03-23 05:35:39.697942",
"item": "quay.io/app-sre/grafana:6.7.4",
"rc": 125,
"start": "2021-03-23 05:35:38.981880"
}
STDERR:
Trying to pull registry.access.redhat.com/grafana:6.7.4...
name unknown: Repo not found
Trying to pull registry.redhat.io/grafana:6.7.4...
unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication
Trying to pull docker.io/library/grafana:6.7.4...
denied: requested access to the resource is denied
Error: error pulling image "grafana:6.7.4": unable to pull grafana:6.7.4: 3 errors occurred:
* Error initializing source docker://registry.access.redhat.com/grafana:6.7.4: Error reading manifest 6.7.4 in registry.access.redhat.com/grafana: name unknown: Repo not found
* Error initializing source docker://registry.redhat.io/grafana:6.7.4: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication
* Error initializing source docker://grafana:6.7.4: Error reading manifest 6.7.4 in docker.io/library/grafana: errors:
denied: requested access to the resource is denied
unauthorized: authentication required
MSG:
non-zero return code
...ignoring
TASK [container-build : List down missing containers] **************************
Tuesday 23 March 2021 05:35:39 +0000 (0:00:01.208) 0:46:07.278 *********
ok: [undercloud] => (item={'msg': 'non-zero return code', 'cmd': ['podman', 'pull', 'grafana:6.7.4'], 'stdout': '', 'stderr': 'Trying to pull registry.access.redhat.com/grafana:6.7.4...\n name unknown: Repo not found\nTrying to pull registry.redhat.io/grafana:6.7.4...\n unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication\nTrying to pull docker.io/library/grafana:6.7.4...\n denied: requested access to the resource is denied\nError: error pulling image "grafana:6.7.4": unable to pull grafana:6.7.4: 3 errors occurred:\n\t* Error initializing source docker://registry.access.redhat.com/grafana:6.7.4: Error reading manifest 6.7.4 in registry.access.redhat.com/grafana: name unknown: Repo not found\n\t* Error initializing source docker://registry.redhat.io/grafana:6.7.4: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication\n\t* Error initializing source docker://grafana:6.7.4: Error reading manifest 6.7.4 in docker.io/library/grafana: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required', 'rc': 125, 'start': '2021-03-23 05:35:38.981880', 'end': '2021-03-23 05:35:39.697942', 'delta': '0:00:00.716062', 'changed': True, 'failed': True, 'invocation': {'module_args': {'_raw_params': 'podman pull grafana:6.7.4', 'warn': False, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [], 'stderr_lines': ['Trying to pull registry.access.redhat.com/grafana:6.7.4...', ' name unknown: Repo not found', 'Trying to pull registry.redhat.io/grafana:6.7.4...', ' unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication', 'Trying to pull docker.io/library/grafana:6.7.4...', ' denied: requested access to the resource is denied', 'Error: error pulling image "grafana:6.7.4": unable to pull grafana:6.7.4: 3 errors occurred:', '\t* Error initializing source docker://registry.access.redhat.com/grafana:6.7.4: Error reading manifest 6.7.4 in registry.access.redhat.com/grafana: name unknown: Repo not found', '\t* Error initializing source docker://registry.redhat.io/grafana:6.7.4: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication', '\t* Error initializing source docker://grafana:6.7.4: Error reading manifest 6.7.4 in docker.io/library/grafana: errors:', 'denied: requested access to the resource is denied', 'unauthorized: authentication required'], 'item': 'quay.io/app-sre/grafana:6.7.4', 'ansible_loop_var': 'item'})
TASK [container-build : Fail the job if missing containers exists] *************
Tuesday 23 March 2021 05:35:39 +0000 (0:00:00.131) 0:46:07.409 *********
fatal: [undercloud]: FAILED! => {
"changed": false
}
MSG:
Failed to pull following containers: ['quay.io/app-sre/grafana:6.7.4']
```
that image is maintained by different team with in organization which we donot have control.
In order to fix it, we need to pull container images from docker.io to quay.io/tripleo.
Here is the series of patches proposed to mitigate the issue; https:/