Failed to create a build container

Bug #1996940 reported by Scott Little
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Low
Scott Little

Bug Description

Brief Description
-----------------
stx-init-env --rebuild sometimes fails due to issues connecting to various software source that contribute to building of the build containers. This seems to be much more common in the early morning hours (~2 am EST).

Ideally, CENGN has a mirror of all of these inputs. Can we modify the Dockerfiles to pull from CENGN if the primary source is down or unreachable?

Severity
--------
Minor

Steps to Reproduce
------------------
stx-init-env --rebuild

Expected Behavior
------------------
Build containers created successfully

Actual Behavior
----------------
Failure to create build containers

Reproducibility
---------------
Intermittent, but rare <1%

System Configuration
--------------------
N/A

Branch/Pull Time/Commit
-----------------------
20221116T070000Z

Last Pass
---------
20221115T070000Z

Timestamp/Logs
--------------

02:15:38 + ./stx-init-env --rebuild
...
02:18:25 Successfully tagged stx-builder:jenkins-debian-master-v0.1.0
02:18:25 built image stx-builder:jenkins-debian-master-v0.1.0
02:18:25 Sending build context to Docker daemon 317.4kB

02:18:26 Step 1/19 : FROM debian:bullseye
...
02:20:35 Step 5/19 : RUN cd /tmp && wget http://snapshot.debian.org/archive/debian/20211128T160803Z/pool/main/d/debootstrap/debootstrap_1.0.126%2Bnmu1_all.deb && dpkg -i debootstrap_1.0.126+nmu1_all.deb
02:20:35 ---> Running in aa077b054066
02:20:35 --2022-11-16 07:20:32-- http://snapshot.debian.org/archive/debian/20211128T160803Z/pool/main/d/debootstrap/debootstrap_1.0.126%2Bnmu1_all.deb
02:20:35 Resolving snapshot.debian.org (snapshot.debian.org)... 193.62.202.27, 185.17.185.185, 2001:630:206:4000:1a1a:0:c13e:ca1b, ...
02:20:35 Connecting to snapshot.debian.org (snapshot.debian.org)|193.62.202.27|:80... connected.
02:21:43 HTTP request sent, awaiting response... 503 Backend fetch failed
02:21:43 2022-11-16 07:21:32 ERROR 503: Backend fetch failed.
02:21:43
02:21:43 The command '/bin/sh -c cd /tmp && wget http://snapshot.debian.org/archive/debian/20211128T160803Z/pool/main/d/debootstrap/debootstrap_1.0.126%2Bnmu1_all.deb && dpkg -i debootstrap_1.0.126+nmu1_all.deb' returned a non-zero code: 8

Test Activity
-------------
Build

Workaround
----------
retry stx-init-env after waiting for connectivity to the upstream source to been restored.

Scott Little (slittle1)
summary: - Failed to creat a build container
+ Failed to create a build container
Ghada Khalil (gkhalil)
Changed in starlingx:
importance: Undecided → Low
tags: added: stx.build
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tools (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/tools/+/934998

Changed in starlingx:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tools (master)

Reviewed: https://review.opendev.org/c/starlingx/tools/+/934998
Committed: https://opendev.org/starlingx/tools/commit/b9e695f8f7a1472c87a3661412a39ec7802c33e1
Submitter: "Zuul (22348)"
Branch: master

commit b9e695f8f7a1472c87a3661412a39ec7802c33e1
Author: Scott Little <email address hidden>
Date: Wed Nov 13 13:40:32 2024 -0500

    fix: stx-pkgbuilder failed to build due to snapshot.debian.org 503

    Upversion from debootstrap_1.0.126+nmu1_all.deb
    to debootstrap_1.0.128+nmu2+deb12u1_all.deb.

    The new version is part of snapshot.debian.org/archive/debian/20211128T160803Z
    And mirror.starlingx.windriver.com has a mirror of that snapshot.
    This in turn allows the stx-pkgbuilder container to pull debootstrap
    from either of two sources.

    Closes-Bug: 1996940
    Change-Id: Ifc5b5d1a4197df8cc9bcae0e1e55c36dcf51173c
    Signed-off-by: Scott Little <email address hidden>

Changed in starlingx:
status: In Progress → Fix Released
Ghada Khalil (gkhalil)
tags: added: stx.10.0
Changed in starlingx:
assignee: nobody → Scott Little (slittle1)
Revision history for this message
Scott Little (slittle1) wrote :

Failed again last night ...
container: stx-pkgbuilder

02:24:55 > [ 5/18] RUN cd /tmp && wget http://snapshot.debian.org/archive/debian/20211128T160803Z/pool/main/d/debootstrap/debootstrap_1.0.126%2Bnmu1_all.deb && dpkg -i debootstrap_1.0.126+nmu1_all.deb:

02:24:55 503 Service Unavailable

Revision history for this message
Scott Little (slittle1) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.