StarlingX

Update build to produce list of docker images required for platform deployment

Bug #1834504 reported by Frank Miller on 2019-06-27

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	StarlingX	In Progress	Low	Abraham Arce

Bug Description

From the starlingx-discuss mailing list the following request was made by Cheng Li but several others also requested this list.

Subject: [Starlingx-discuss] Docker image list

As you know many docker images are pulled during starlingx deployment. It may be fast to pull all these images in America, but it’s very slow in China.
To speed up starlingx deployment, I have set up a private docker registry for which I sync images every night from upstream registries by cron job.
I installed starlingx without using my private docker registry so that I can get the upstream docker image list. Every item in the list is synced every day.

This works fine except that the docker image list changes sometimes. In this case, I would have to collect the image list by deploying without using private docker registry, which is very slow.
So I wonder if it’s possible to publish the docker image list file together with ISO and tarball on CENGN?

--------------

This launchpad will be used to track an enhancement for the build to produce a list of docker images required by StarlingX to deploy the platform (up to the point where the platform-integ-apps is applied).

Note: the images required by the stx-openstack application can be obtained already from the CENGN build output.

Tags:

Revision history for this message

Bob Church (rchurch) wrote on 2019-06-27:

Download full text (5.8 KiB)

We have the following:

Stage 1: After Ansible setup

REPOSITORY TAG
starlingx/k8s-cni-sriov master-centos-stable-latest
quay.io/airshipit/armada dd2e56c473549fd16f94212b553ed58c48d1f51b-ubuntu_bionic
quay.io/calico/node v3.6.2
quay.io/calico/cni v3.6.2
quay.io/calico/kube-controllers v3.6.2
k8s.gcr.io/kube-proxy v1.13.5
k8s.gcr.io/kube-apiserver v1.13.5
k8s.gcr.io/kube-controller-manager v1.13.5
k8s.gcr.io/kube-scheduler v1.13.5
nfvpe/multus v3.2
gcr.io/kubernetes-helm/tiller v2.13.1
k8s.gcr.io/coredns 1.2.6
k8s.gcr.io/pause 3.1

Stage 2: After first unlock. platform-integ-apps are applied.

REPOSITORY TAG
quay.io/external_storage/rbd-provisioner v2.1.1-k8s1.11
port/ceph-config-helper v1.10.3

# For these they all are derived from here:
# starlingx/k8s-cni-sriov master-centos-stable-latest
# quay.io/calico/node v3.6.2
# quay.io/calico/cni v3.6.2
# quay.io/calico/kube-controllers v3.6.2
[rchurch@soho-rchurch-lx2 cgcs-root]$ grep -ri "image:" $MY_REPO/stx/stx-config/puppet-manifests/src/modules/platform/templates | sort -u
/l/d/rchurch/starlingx-2/cgcs-root/stx/stx-config/puppet-manifests/src/modules/platform/templates/calico.yaml.erb: image: "<%= @quay_registry %>/calico/cni:v3.6.2"
/l/d/rchurch/starlingx-2/cgcs-root/stx/stx-config/puppet-manifests/src/modules/platform/templates/calico.yaml.erb: image: "<%= @quay_registry %>/calico/kube-controllers:v3.6.2"
/l/d/rchurch/starlingx-2/cgcs-root/stx/stx-config/puppet-manifests/src/modules/platform/templates/calico.yaml.erb: image: "<%= @quay_registry %>/calico/node:v3.6.2"
/l/d/rchurch/starlingx-2/cgcs-root/stx/stx-config/puppet-manifests/src/modules/platform/templates/multus.yaml.erb: image: <%= @docker_registry %>/nfvpe/multus:v3.2
/l/d/rchurch/starlingx-2/cgcs-root/stx/stx-config/puppet-manifests/src/modules/platform/templates/sriov-cni.yaml.erb: image: <%= @docker_registry %>/starlingx/k8s-cni-sriov:master-centos-stable-latest
/l/d/rchurch/starlingx-2/cgcs-root/stx/stx-config/puppet-manifests/src/modules/platform/templates/sriovdp-daemonset.yaml.erb: image: <%= @docker_registry %>/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest

# For these we can derive it from the Kubernetes version:
# k8s.gcr.io/kube-proxy v1.13.5
# k8s.gcr.io/kube-apiserver v1.13.5
# k8s.gcr.io/kube-controller-manager v1.13.5
# k8s.gcr.io/kube-scheduler v1.13.5
[rchurch@soho-rchurch-lx2 cgcs-root]$ grep -ri "kubernetesVersion" $MY_REPO/stx/stx-config/puppet-...

We have the following:

Stage 1:  After Ansible setup
 
REPOSITORY                           TAG                                                     
starlingx/k8s-cni-sriov              master-centos-stable-latest
quay.io/airshipit/armada             dd2e56c473549fd16f94212b553ed58c48d1f51b-ubuntu_bionic
quay.io/calico/node                  v3.6.2
quay.io/calico/cni                   v3.6.2
quay.io/calico/kube-controllers      v3.6.2
k8s.gcr.io/kube-proxy                v1.13.5
k8s.gcr.io/kube-apiserver            v1.13.5
k8s.gcr.io/kube-controller-manager   v1.13.5
k8s.gcr.io/kube-scheduler            v1.13.5
nfvpe/multus                         v3.2
gcr.io/kubernetes-helm/tiller        v2.13.1
k8s.gcr.io/coredns                   1.2.6
k8s.gcr.io/pause                     3.1
 
 
Stage 2:  After first unlock.   platform-integ-apps are applied.
 
REPOSITORY                           TAG                                                     
quay.io/external_storage/rbd-provisioner                       v2.1.1-k8s1.11
port/ceph-config-helper                                        v1.10.3

I took a crack at trying to locate the information from my private build. We would basically need a set of greps to derive them. I think it can be done if someone wants to take on packaging it into a script. 
 
 
# For these they all are derived from here:
# starlingx/k8s-cni-sriov              master-centos-stable-latest
# quay.io/calico/node                  v3.6.2
# quay.io/calico/cni                   v3.6.2
# quay.io/calico/kube-controllers      v3.6.2
[rchurch@soho-rchurch-lx2 cgcs-root]$ grep -ri "image:" $MY_REPO/stx/stx-config/puppet-manifests/src/modules/platform/templates | sort -u
/l/d/rchurch/starlingx-2/cgcs-root/stx/stx-config/puppet-manifests/src/modules/platform/templates/calico.yaml.erb:          image: "<%= @quay_registry %>/calico/cni:v3.6.2"
/l/d/rchurch/starlingx-2/cgcs-root/stx/stx-config/puppet-manifests/src/modules/platform/templates/calico.yaml.erb:          image: "<%= @quay_registry %>/calico/kube-controllers:v3.6.2"
/l/d/rchurch/starlingx-2/cgcs-root/stx/stx-config/puppet-manifests/src/modules/platform/templates/calico.yaml.erb:          image: "<%= @quay_registry %>/calico/node:v3.6.2"
/l/d/rchurch/starlingx-2/cgcs-root/stx/stx-config/puppet-manifests/src/modules/platform/templates/multus.yaml.erb:        image: <%= @docker_registry %>/nfvpe/multus:v3.2
/l/d/rchurch/starlingx-2/cgcs-root/stx/stx-config/puppet-manifests/src/modules/platform/templates/sriov-cni.yaml.erb:        image: <%= @docker_registry %>/starlingx/k8s-cni-sriov:master-centos-stable-latest
/l/d/rchurch/starlingx-2/cgcs-root/stx/stx-config/puppet-manifests/src/modules/platform/templates/sriovdp-daemonset.yaml.erb:        image: <%= @docker_registry %>/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest
 
# For these we can derive it from the Kubernetes version:
# k8s.gcr.io/kube-proxy                v1.13.5
# k8s.gcr.io/kube-apiserver            v1.13.5
# k8s.gcr.io/kube-controller-manager   v1.13.5
# k8s.gcr.io/kube-scheduler            v1.13.5
[rchurch@soho-rchurch-lx2 cgcs-root]$ grep -ri "kubernetesVersion" $MY_REPO/stx/stx-config/puppet-manifests/src/modules/platform/templates | sort -u
/l/d/rchurch/starlingx-2/cgcs-root/stx/stx-config/puppet-manifests/src/modules/platform/templates/kubeadm.yaml.erb:kubernetesVersion: 1.13.5

# k8s.gcr.io/coredns                   1.2.6
# k8s.gcr.io/pause                     3.1
These are baked into the kubeadm and are pulled when it’s initialized. We can see what versions that it will pull if left to its own devices:
 
controller-0:/etc# kubeadm config images list
I0624 20:09:40.825976  572565 version.go:237] remote version is much newer: v1.15.0; falling back to: stable-1.13
k8s.gcr.io/kube-apiserver:v1.13.7
k8s.gcr.io/kube-controller-manager:v1.13.7
k8s.gcr.io/kube-scheduler:v1.13.7
k8s.gcr.io/kube-proxy:v1.13.7
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.2.24
k8s.gcr.io/coredns:1.2.6

When we set the kubernetesVersion in the ERB file we could also document the versions of pause and coredns in a comment that we could use with an extraction script.

# For these helm.pp:
# quay.io/airshipit/armada             af8a9ffd0873c2fbc915794e235dbd357f2adab1
# gcr.io/kubernetes-helm/tiller        v2.13.1
[rchurch@soho-rchurch-lx2 cgcs-root]$ grep -ri "pull.*airshipit" $MY_REPO/stx/stx-config/puppet-manifests/src/modules/platform/manifests | sort -u
/l/d/rchurch/starlingx-2/cgcs-root/stx/stx-config/puppet-manifests/src/modules/platform/manifests/helm.pp:        command   => "docker image pull ${quay_registry}/airshipit/armada:af8a9ffd0873c2fbc915794e235dbd357f2adab1",
[rchurch@soho-rchurch-lx2 cgcs-root]$ grep -ri "pull.*tiller" $MY_REPO/stx/stx-config/puppet-manifests/src/modules/platform/manifests | sort -u
/l/d/rchurch/starlingx-2/cgcs-root/stx/stx-config/puppet-manifests/src/modules/platform/manifests/helm.pp:        command   => "docker image pull ${gcr_registry}/kubernetes-helm/tiller:v2.13.1",
 
# For platform-integ-apps: These can be moved and defined in $MY_REPO/stx/stx-config/kubernetes/platform/stx-platform/stx-platform-helm/stx-platform-helm/manifests/manifest.yaml for a single grep:
# quay.io/external_storage/rbd-provisioner                       v2.1.1-k8s1.11
# port/ceph-config-helper                                        v1.10.3
[rchurch@soho-rchurch-lx2 starlingx-2]$ grep -ri rbd-provisioner: $MY_REPO/stx/stx-config/kubernetes/helm-charts/rbd-provisioner
/l/d/rchurch/starlingx-2/cgcs-root/stx/stx-config/kubernetes/helm-charts/rbd-provisioner/values.yaml:    rbd_provisioner: quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11
[rchurch@soho-rchurch-lx2 cgcs-root]$ grep -ri ceph-config-helper: $MY_REPO/stx/stx-config/kubernetes/helm-charts/ceph-pools-audit
/l/d/rchurch/starlingx-2/cgcs-root/stx/stx-config/kubernetes/helm-charts/ceph-pools-audit/values.yaml:    ceph_config_helper: docker.io/port/ceph-config-helper:v1.10.3

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2019-06-28:

Marking as not gating - this is more a build/tools enhancement

Changed in starlingx:
importance:	Undecided → Low
status:	New → Triaged
tags:	added: stx.build stx.helpwanted stx.tools
Changed in starlingx:
assignee:	nobody → Abraham Arce (xe1gyq)

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2019-06-28:

Assigning to the build/tools team

OpenStack Infra (hudson-openstack) on 2023-06-02

Changed in starlingx:
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-08-30: Fix merged to docs (master)

Reviewed: https://review.opendev.org/c/starlingx/docs/+/884643
Committed: https://opendev.org/starlingx/docs/commit/1fc795a68f166607dfd7f75d4c74eebea8756b1f
Submitter: "Zuul (22348)"
Branch: master

commit 1fc795a68f166607dfd7f75d4c74eebea8756b1f
Author: Bruno Muniz <email address hidden>
Date: Mon May 29 17:23:16 2023 -0300

Add a local registry guide for developers

    The guide is aimed at developers working in virtual environments that
    need to build or run several controller-0 setups (specifically, to
    developers in need of running the Ansible bootstrap playbook multiple
    times).

    By configuring a local registry and downloading the needed images
    beforehand, all subsequent installations can point to this local
    registry to download images from it to the controller-0. Not going to
    the internet everytime, devs are not subject to public registries' rate
    limits.

    Story: 2005051
    Task: 47895
    Partial-Bug: 1834504
    Partial-Bug: 1888610

Change-Id: Ie3e3cbc09030a6491f7b573a4ea5fd57e3158d54
Signed-off-by: Bruno Muniz <email address hidden>

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.