tripleo

tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates is failing while pulling containers

Bug #1921285 reported by wes hayutin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Incomplete
Critical
Unassigned
tripleo xena-3

Bug Description

https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates

tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates openstack/tripleo-heat-templates master check 758811,8 40 mins 35 secs 2021-03-24 18:54:03 FAILURE
tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates openstack/tripleo-heat-templates master check 782288,1 33 mins 37 secs 2021-03-24 18:26:11 FAILURE
tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates openstack/tripleo-heat-templates master gate 782302,1 36 mins 10 secs 2021-03-24 17:19:03 FAILURE
tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates openstack/tripleo-heat-templates master check 746487,1 34 mins 12 secs 2021-03-24 16:51:03 FAILURE
tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates openstack/tripleo-heat-templates master check 779801,3 34 mins 38 secs 2021-03-24 16:11:56 FAILURE
tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates openstack/tripleo-heat-templates master gate 781069,2 32 mins 35 secs 2021-03-24 15:30:50 FAILURE

2021-03-24 19:27:02.047599 | fa163ecb-39d5-6d34-76ae-000000000cf2 | FATAL | Container image prepare | undercloud | error={"changed": false, "error": "Not found image: http://149.202.163.180:5001/v2/tripleomaster/openstack-neutron-server/manifests/ec263196cc051357ad315aa33ed62b26", "msg": "Error running container image prepare: Not found image: http://149.202.163.180:5001/v2/tripleomaster/openstack-neutron-server/manifests/ec263196cc051357ad315aa33ed62b26", "params": {}, "success": false}

2021-03-24 18:53:17.101377 | fa163e40-8eea-853f-4e40-000000000cf2 | FATAL | Container image prepare | undercloud | error={"changed": false, "error": "Not found image: http://198.72.124.68:5001/v2/tripleomaster/openstack-rsyslog/manifests/ec263196cc051357ad315aa33ed62b26", "msg": "Error running container image prepare: Not found image: http://198.72.124.68:5001/v2/tripleomaster/openstack-rsyslog/manifests/ec263196cc051357ad315aa33ed62b26", "params": {}, "success": false}

021-03-24 17:46:45.453681 | fa163e7d-917d-8779-da5b-000000000cf2 | FATAL | Container image prepare | undercloud | error={"changed": false, "error": "Not found image: http://198.72.124.66:5001/v2/tripleomaster/openstack-keystone/manifests/ec263196cc051357ad315aa33ed62b26", "msg": "Error running container image prepare: Not found image: http://198.72.124.66:5001/v2/tripleomaster/openstack-keystone/manifests/ec263196cc051357ad315aa33ed62b26", "params": {}, "success": false}

Revision history for this message
wes hayutin (weshayutin) wrote :

master promoted today, may be related

Revision history for this message
wes hayutin (weshayutin) wrote :

https://review.opendev.org/q/topic:%22container-updates%22+(status:open%20OR%20status:merged)

wes hayutin (weshayutin)
Changed in tripleo:
status: Triaged → Incomplete
Marios Andreou (marios-b)
Changed in tripleo:
milestone: wallaby-rc1 → xena-1
Marios Andreou (marios-b)
Changed in tripleo:
milestone: xena-1 → xena-2
Marios Andreou (marios-b)
Changed in tripleo:
milestone: xena-2 → xena-3
Revision history for this message
Juan Badia Payno (jbadiapa) wrote :
Download full text (3.8 KiB)

Pasted part of the logs in case they are deleted.

I think we hit this issue again at https://zuul.opendev.org/t/openstack/build/eec4b913fae44df6a33dfc2ee0442a2f/log/logs/undercloud/home/zuul/undercloud_install.log#1668

=====
2021-08-18 14:24:29.215817 | bc764e20-0d31-a0c7-a66d-000000000de1 | TASK | Container image prepare

2021-08-18 14:24:31.130811 | bc764e20-0d31-a0c7-a66d-000000000de1 | FATAL | Container image prepare | undercloud | error={"changed": false, "error": "Not found image: http://104.130.169.80:5001/v2/tripleomaster/openstack-neutron-server/manifests/a085460a23443cfc28a1dc6c667e110e", "msg": "Error running container image prepare: Not found image: http://104.130.169.80:5001/v2/tripleomaster/openstack-neutron-server/manifests/a085460a23443cfc28a1dc6c667e110e", "params": {}, "success": false}

2021-08-18 14:24:31.133750 | bc764e20-0d31-a0c7-a66d-000000000de1 | TIMING | tripleo_container_image_prepare : Container image prepare | undercloud | 0:03:03.199511 | 1.92s

PLAY RECAP *********************************************************************

localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0

undercloud : ok=209 changed=110 unreachable=0 failed=1 skipped=36 rescued=0 ignored=2
======

There are more logs at https://zuul.opendev.org/t/openstack/build/eec4b913fae44df6a33dfc2ee0442a2f/log/logs/undercloud/var/log/tripleo-container-image-prepare.log

=====
2021-08-18 14:24:30,943 53106 DEBUG urllib3.connectionpool [ ] http://192.168.24.1:8787 "POST /v2/tripleomaster/openstack-neutron-server/blobs/uploads/ HTTP/1.1" 404 196

2021-08-18 14:24:30,944 53106 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id 47c1dea99076185243b76070ac68d51264e40eea, status 404, reason Not Found, text <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

<html><head>

<title>404 Not Found</title>

</head><body>

<h1>Not Found</h1>

<p>The requested URL was not found on this server.</p>

</body></html>

2021-08-18 14:24:30,944 53111 DEBUG urllib3.connectionpool [ ] http://192.168.24.1:8787 "GET /v2/ HTTP/1.1" 200 2

2021-08-18 14:24:30,944 53106 DEBUG urllib3.util.retry [ ] Converted retries value: 8 -> Retry(total=8, connect=None, read=None, redirect=None, status=None)

2021-08-18 14:24:30,945 53107 DEBUG urllib3.connectionpool [ ] http://192.168.24.1:8787 "POST /v2/tripleomaster/openstack-ironic-inspector/blobs/uploads/ HTTP/1.1" 404 196
..
.
.
..
2021-08-18 14:24:30,960 53109 DEBUG urllib3.connectionpool [ ] http://104.130.169.80:5001 "GET /v2/tripleomaster/openstack-ironic-conductor/manifests/a085460a23443cfc28a1dc6c667e110e HTTP/1.1" 404 122

2021-08-18 14:24:30,960 53107 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id 75b572ca18e9d1a2cf0765664e0571517b1b0da6, status 404, reason Not Found, text {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"a085460a23443cfc28a1dc6c667e110e"}}]}

2021-08-18 14:24:30,960 53106 DEBUG urllib3.connectionpool [ ] http://104.130.169.80:5001 "GET /v2/tripleomaster/openstack-neutron-server/manifests/a085460a23443cfc28a1dc6c667e110e HTTP/1.1" 404 122

2...

Read more...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.