tripleo

cs8 octavia/cloudops/tempest/common/network/tripleo/validation wallaby fails with "Error running container image prepare"

Bug #1978319 reported by Soniya Murlidhar Vyas
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Ananya Banerjee
tripleo zed-1

Bug Description

cs8 octavia wallaby fails with "Error running container image prepare"

Following is the traceback observed:-

[WARNING]: ('undercloud -> localhost', 'fa163e1f-b116-711c-f8a6-0000000000a0')
missing from stats
2022-06-09 13:55:29.266349 | fa163e1f-b116-711c-f8a6-0000000000a1 | TIMING | include_tasks | undercloud | 0:03:06.258828 | 0.02s
2022-06-09 13:55:29.289252 | 422c840a-0d1e-435b-bea6-a317c700977e | INCLUDED | /home/zuul/tripleo-deploy/standalone-ansible-uevowtdd/external_deploy_steps_tasks_step1.yaml | undercloud
2022-06-09 13:55:29.305694 | fa163e1f-b116-711c-f8a6-000000001448 | TASK | Container image prepare
2022-06-09 13:56:30.988720 | fa163e1f-b116-711c-f8a6-000000001448 | FATAL | Container image prepare | undercloud | error={"changed": false, "error": "401 Client Error: UNAUTHORIZED for url: https://quay.rdoproject.org/v2/tripleowallaby/openstack-rsyslog/manifests/376a92910b42e0e0f03dd304dabb7a60", "msg": "Error running container image prepare: 401 Client Error: UNAUTHORIZED for url: https://quay.rdoproject.org/v2/tripleowallaby/openstack-rsyslog/manifests/376a92910b42e0e0f03dd304dabb7a60", "params": {}, "success": false}
2022-06-09 13:56:30.992629 | fa163e1f-b116-711c-f8a6-000000001448 | TIMING | tripleo_container_image_prepare : Container image prepare | undercloud | 0:04:07.985113 | 61.69s

PLAY RECAP *********************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0
standalone : ok=248 changed=137 unreachable=0 failed=0 skipped=79 rescued=0 ignored=0
undercloud : ok=18 changed=6 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

Please refer following links for more details:

1. https://logserver.rdoproject.org/openstack-component-octavia/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-octavia-wallaby/0fbf8ef/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz
2. https://logserver.rdoproject.org/openstack-component-octavia/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-octavia-wallaby/4219fe6/logs/undercloud/home/zuul/ansible.log.txt.gz
3. https://logserver.rdoproject.org/openstack-component-octavia/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-octavia-wallaby/12d1999/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

Revision history for this message
Ronelle Landy (rlandy) wrote :

https://logserver.rdoproject.org/openstack-component-octavia/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-octavia-wallaby/0fbf8ef/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz

Revision history for this message
Soniya Murlidhar Vyas (svyas) wrote :

Following is the traceback observed in [1]:

2022-06-09 13:56:05,169 67973 WARNING tripleo_common.image.image_uploader [ ] Re-authenticating: id af32f8507626bcfa8e489a7cd1826321188d4919, count 10
2022-06-09 13:56:05,172 67973 DEBUG urllib3.connectionpool [ ] Resetting dropped connection: quay.rdoproject.org
2022-06-09 13:56:05,191 67973 DEBUG urllib3.connectionpool [ ] https://quay.rdoproject.org:443 "GET /v2/ HTTP/1.1" 401 4
2022-06-09 13:56:05,191 67973 DEBUG tripleo_common.image.image_uploader [ ] https://quay.rdoproject.org/v2/ status code 401
2022-06-09 13:56:05,192 67973 DEBUG tripleo_common.image.image_uploader [ ] Using bearer token auth
2022-06-09 13:56:05,194 67973 DEBUG urllib3.connectionpool [ ] Resetting dropped connection: quay.rdoproject.org
2022-06-09 13:56:05,225 67973 DEBUG urllib3.connectionpool [ ] https://quay.rdoproject.org:443 "GET /v2/auth?service=quay.rdoproject.org&scope=repository%3Atripleowallaby%2Fopenstack-neutron-metadata-agent-ovn%3Apull HTTP/1.1" 200 937
2022-06-09 13:56:05,227 67973 DEBUG tripleo_common.image.image_uploader [ ] Session authenticated: id b1e1aba07925810b12391e9cfc601c7482eefc6b
2022-06-09 13:56:05,227 67973 ERROR tripleo_common.image.image_uploader [ ] [192.168.24.1:8787/tripleowallaby/openstack-neutron-metadata-agent-ovn:376a92910b42e0e0f03dd304dabb7a60-updated-20220609134036] Failed uploading the target image
2022-06-09 13:56:05,231 67973 INFO tripleo_common.image.image_uploader [ ] [quay.rdoproject.org/tripleowallaby/openstack-swift-account:376a92910b42e0e0f03dd304dabb7a60] Starting upload image process
2022-06-09 13:56:05,232 67973 DEBUG urllib3.util.retry [ ] Converted retries value: 8 -> Retry(total=8, connect=None, read=None, redirect=None, status=None)
2022-06-09 13:56:05,234 67973 DEBUG urllib3.connectionpool [ ] Starting new HTTP connection (1): 192.168.24.1:8787
2022-06-09 13:56:05,235 67973 DEBUG urllib3.connectionpool [ ] http://192.168.24.1:8787 "GET /v2/ HTTP/1.1" 200 2
2022-06-09 13:56:05,236 67973 DEBUG tripleo_common.image.image_uploader [ ] http://192.168.24.1:8787/v2/ status code 200
2022-06-09 13:56:05,238 67973 DEBUG urllib3.connectionpool [ ] http://192.168.24.1:8787 "GET /v2/tripleowallaby/openstack-swift-account/tags/list HTTP/1.1" 404 196
2022-06-09 13:56:05,238 67973 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id 2bee4f764201101ad89d85b0beaf3149be7af142, status 404, reason Not Found, text

From the above traceback we can see that image has failed to get uploaded.

[1] https://logserver.rdoproject.org/openstack-component-octavia/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-octavia-wallaby/0fbf8ef/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz
[2] https://logserver.rdoproject.org/openstack-component-octavia/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-octavia-wallaby/4219fe6/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz

Revision history for this message
Rabi Mishra (rabi) wrote :

> 401 Client Error: UNAUTHORIZED for url: https://quay.rdoproject.org/v2/tripleowallaby/openstack-glance-api/manifests/376a92910b42e0e0f03dd304dabb7a60

Namespace looks incorrect , should be https://quay.rdoproject.org/v2/tripleowallabycentos8 I guess, probably need to update[1].

[1] https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release/centosci/CentOS-8/wallaby-current-tripleo.yml#L12

Revision history for this message
Soniya Murlidhar Vyas (svyas) wrote :

this issue is also seen centos-8 standalone cloudops wallaby - https://logserver.rdoproject.org/openstack-component-cloudops/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-cloudops-wallaby/eabc877/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz

https://logserver.rdoproject.org/openstack-component-cloudops/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-cloudops-wallaby/3df7283/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz

Revision history for this message
Soniya Murlidhar Vyas (svyas) wrote (last edit ):

same is happening with c8 standalone tempest wallaby and fs01 tripleo wallaby -

1. https://logserver.rdoproject.org/openstack-component-tempest/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-tempest-wallaby/eb4645d/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz

2. https://logserver.rdoproject.org/openstack-component-tempest/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-tempest-wallaby/d85d060/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz

3. https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-wallaby/f4cc084/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz

Soniya Murlidhar Vyas (svyas)
summary: - cs8 octavia wallaby fails with "Error running container image prepare"
+ cs8 octavia/cloudops/tempest/common/network/tripleo/validation wallaby
+ fails with "Error running container image prepare"
Revision history for this message
Ronelle Landy (rlandy) wrote :

https://review.opendev.org/c/openstack/tripleo-quickstart/+/843007/ may fix this. Patch is in gates right now

Revision history for this message
Ronelle Landy (rlandy) wrote :

Trying testproject https://review.rdoproject.org/r/c/testproject/+/40101 with depends-on: https://review.opendev.org/c/openstack/tripleo-quickstart/+/843007/

Revision history for this message
Dariusz Smigiel (smigiel-dariusz) wrote :

After testing it with depends-on: https://review.opendev.org/c/openstack/tripleo-quickstart/+/843007/ it's still failing.

Breaking change: https://github.com/openstack/tripleo-ci/commit/a70a36db14e73008f5cfef17178170f2fbbc9b33

Changed in tripleo:
assignee: nobody → Ananya Banerjee (frenzyfriday)
Revision history for this message
Ronelle Landy (rlandy) wrote :

@ananya: component jobs are trying to pull containers from the RDO quay registry with the namespace from the quay.io registry

Changed in tripleo:
milestone: none → zed-1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-quickstart (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/845568

Revision history for this message
Ananya Banerjee (frenzyfriday) wrote :

Looks like wallaby c8 component jobs pass if we change this https://review.opendev.org/c/openstack/tripleo-quickstart/+/845568

But isnt https://review.opendev.org/c/openstack/tripleo-quickstart/+/845568/1/config/release/tripleo-ci/CentOS-8/wallaby.yml#26 supposed to be the namespace in quay.io, as we are still using quay.io and not rdoproj for the check jobs https://review.opendev.org/c/openstack/tripleo-quickstart/+/845568/1/config/release/tripleo-ci/CentOS-8/wallaby.yml#24 ?

test proj: https://review.rdoproject.org/r/c/testproject/+/40101

Revision history for this message
Soniya Murlidhar Vyas (svyas) wrote :

On wallaby/Cloudops/tempest, we are green[1],

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-quickstart (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/845568
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/12fc4167c70531da80403746c1f172ec56570679
Submitter: "Zuul (22348)"
Branch: master

commit 12fc4167c70531da80403746c1f172ec56570679
Author: frenzyfriday <email address hidden>
Date: Mon Jun 13 10:40:41 2022 +0200

    Changing the namespace for wallabyc8 to tripleowallabycentos8 only for component jobs

    The file wallaby.yml is used both by component and check jobs. We need to change the namespace to tripleowallabycentos8 only for the component jobs as they use registry quay.rdoproj [1] [2]

    [1] https://opendev.org/openstack/tripleo-ci/src/branch/master/toci-quickstart/config/testenv/multinode-vexxhost.yml#L53
    [2] https://opendev.org/openstack/tripleo-ci/src/branch/master/toci-quickstart/config/testenv/ovb-vexxhost.yml#L39

    Related-Bug: 1978319

    Testproj: https://review.rdoproject.org/r/c/testproject/+/43594

    Change-Id: I2fc2bea2b27ea688f68d3c96eb3f7e5d29405e33

Ronelle Landy (rlandy)
Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.