containers-multinode unable to pull images

no longer seeing this issue

I'm hitting this bug too. Basically any release from stein, master, master-tripleo-ci deployed by quickstart is failing by following for me:

"Trying to pull 192.168.24.1:8787/tripleomaster/centos-binary-nova-libvirt:current-tripleo...",
        "Copying blob sha256:aa6f1da0ce62d77237d5b4ba56c92cc8c50e224276c12878da7ff0be38a37b98",
        "Copying blob sha256:e5f193924581b7ed972f249f3bcfad70325d847e0e6edfff704b687e0fc34f15",
        "Copying blob sha256:ee9a79ad1e1013a5b0b5eb95737934a4efcd46daad5d3327479e355996823908",
        "Copying blob sha256:bec77c7a2e0d98b90e1abec7c2b50b3e73afa0e03de630af41aa8a3bd9418ce2",
        "Copying blob sha256:029c9bd93a422fc31a4bf6e3a88d9f7fe00d3d52292e9df7a7f63a3f720d02bd",
        " Digest did not match, expected sha256:ee9a79ad1e1013a5b0b5eb95737934a4efcd46daad5d3327479e355996823908, got sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
        "Error: unable to pull 192.168.24.1:8787/tripleomaster/centos-binary-nova-libvirt:current-tripleo: unable to pull image: Error reading blob sha256:ee9a79ad1e1013a5b0b5eb95737934a4efcd46daad5d3327479e355996823908: Digest did not match, expected sha256:ee9a79ad1e1013a5b0b5eb95737934a4efcd46daad5d3327479e355996823908, got sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"

seeing if we can reproduce outside of CI w/
./quickstart.sh -w /var/tmp/RECREATE_tq/ --clean --tags all -R tripleo-ci/CentOS-7/master whayutin-testbox

./quickstart.sh -w /var/tmp/RECREATE_tq/ --clean --tags all -R tripleo-ci/CentOS-7/master whayutin-testbox

https://pasted.tech/pastes/6170336054fac0bbe4926fc469ff87823984df07
https://paste.fedoraproject.org/paste/TyEixPe0z2shWxVb8kKAhw/
https://paste.fedoraproject.org/paste/wGEG0aKPiG8G8q-DjRR9qw/

must be some proxy getting in your way.

any more details you can provide Martin?

Running the image prepare fails on libvirt image:
(undercloud) [stack@undercloud ~]$ sudo /usr/bin/tripleo-container-image-prepare --roles-file /tmp/ansible.uJ8hgm-role-data --environment-file /home/stack/containers-prepare-parameter.yaml --cleanup partial --log-file fuckin.log --debug
<snip>
019-10-11 13:29:44 740504 ERROR tripleo_common.image.image_export [ ] [tripleomaster/centos-binary-nova-libvirt] Write Failure:
2019-10-11 13:29:44 740504 ERROR tripleo_common.image.image_export [ ] [tripleomaster/centos-binary-nova-libvirt] Broken layer found and removed /var/lib/image-serve/v2/tripleomaster/centos-binary-nova-libvirt/blobs/sha256:ee9a79ad1e1013a5b0b5eb95737934a4efcd46daad5d3327479e355996823908.gz
2019-10-11 13:29:44 740504 DEBUG tripleo_common.image.image_uploader [ ] Unlocking layer sha256:ee9a79ad1e1013a5b0b5eb95737934a4efcd46daad5d3327479e355996823908
2019-10-11 13:29:44 740504 DEBUG tripleo_common.image.image_uploader [ ] Starting acquire for lock sha256:ee9a79ad1e1013a5b0b5eb95737934a4efcd46daad5d3327479e355996823908
2019-10-11 13:29:44 740504 DEBUG tripleo_common.image.image_uploader [ ] Acquired for unlock sha256:ee9a79ad1e1013a5b0b5eb95737934a4efcd46daad5d3327479e355996823908
2019-10-11 13:29:44 740504 DEBUG tripleo_common.image.image_uploader [ ] Updated lock info sha256:ee9a79ad1e1013a5b0b5eb95737934a4efcd46daad5d3327479e355996823908
2019-10-11 13:29:44 740504 DEBUG tripleo_common.image.image_uploader [ ] Released lock on layer sha256:ee9a79ad1e1013a5b0b5eb95737934a4efcd46daad5d3327479e355996823908
2019-10-11 13:29:45 740504 WARNING tripleo_common.image.image_uploader [ ] No lock information provided for layer sha256:ee9a79ad1e1013a5b0b5eb95737934a4efcd46daad5d3327479e355996823908
2019-10-11 13:29:45 740504 DEBUG urllib3.connectionpool [ ] Resetting dropped connection: 192.168.24.1
2019-10-11 13:29:45 740504 DEBUG urllib3.connectionpool [ ] http://192.168.24.1:8787 "HEAD /v2/tripleomaster/centos-binary-nova-libvirt/blobs/sha256:ee9a79ad1e1013a5b0b5eb95737934a4efcd46daad5d3327479e355996823908 HTTP/1.1" 404 0
2019-10-11 13:29:45 740504 DEBUG tripleo_common.image.image_uploader [ ] [sha256:ee9a79ad1e1013a5b0b5eb95737934a4efcd46daad5d3327479e355996823908] Uploading layer
2019-10-11 13:29:45 740504 DEBUG tripleo_common.image.image_export [ ] [tripleomaster/centos-binary-nova-libvirt] Export layer to /var/lib/image-serve/v2/tripleomaster/centos-binary-nova-libvirt/blobs/sha256:ee9a79ad1e1013a5b0b5eb95737934a4efcd46daad5d3327479e355996823908.gz
2019-10-11 13:29:45 740504 INFO tripleo_common.image.image_uploader [ ] [/tripleomaster/centos-binary-nova-libvirt] Fetching layer sha256:ee9a79ad1e1013a5b0b5eb95737934a4efcd46daad5d3327479e355996823908 from https://registry-1.docker.io/v2/tripleomaster/centos-binary-nova-libvirt/blobs/sha256:ee9a79ad1e1013a5b0b5eb95737934a4efcd46daad5d3327479e355996823908
2019-10-11 13:29:45 740504 DEBUG urllib3.connectionpool [ ] https://registry-1.docker.io:443 "GET /v2/tripleomaster/centos-binary-nova-libvirt/blobs/sha256:ee9a79ad1e1013a5b0b5eb95737934a4efcd46daad5d3327479e355996823908 HTTP/1.1" 307 0
2019-10-11 13:29:45 740504 DEBUG urllib3.connectionpool [ ] https://production.cloudflar...

There is not much to say. Reproducer is:

1. Clone tripleo-quickstart
2. Apply following patch, without it on CentOS7 quicktart fails with suntax error:
[root@dell-t5810ws-rdo-13 tripleo-quickstart]$ git show bfbc1d00e2fc82e44e3fdd92382b0ec2d8d13b4c
commit bfbc1d00e2fc82e44e3fdd92382b0ec2d8d13b4c
Author: root <email address hidden>
Date: Mon Oct 7 16:35:11 2019 +0200

    para fix

diff --git a/requirements.txt b/requirements.txt
index 2740482..9206b46 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,6 +1,6 @@
 cmd2==0.8.5
 ara<1.0
-ansible>=2.8,<2.9
+ansible>=2.7,<2.8
 jmespath
 netaddr>=0.7.18
 os-client-config
[root@dell-t5810ws-rdo-13 tripleo-quickstart]$
3. deploy undercloud with:
[root@dell-t5810ws-rdo-13 tripleo-quickstart]$ ./quickstart.sh --clean --teardown all --no-clone --release master-tripleo-ci --nodes /root/tripleo/tripleo-quickstart/para-standard-nodes.yaml dell-t5810ws-rdo-13.tpb.lab.eng.brq.redhat.com
3a. nodes setting:
[root@dell-t5810ws-rdo-13 tripleo-quickstart]$ cat para-standard-nodes.yaml
# Tell tripleo which nodes to deploy.
topology_map:
  Controller:
    scale: 3
  Compute:
    scale: 1

repo_setup_run_update: false
control_memory: 6144
compute_memory: 6144

# undercloud resources
undercloud_disk: 150
#undercloud_memory: 16384
undercloud_memory: 8192
undercloud_vcpu: 4

# opstools resources
opstools_memory: 4096

flavors:
  compute:
    memory: '{{compute_memory|default(default_memory)}}'
    disk: '{{compute_disk|default(default_disk)}}'
    vcpu: '{{compute_vcpu|default(default_vcpu)}}'

  control:
    memory: '{{control_memory|default(default_memory)}}'
    disk: '{{control_disk|default(default_disk)}}'
    vcpu: '{{control_vcpu|default(default_vcpu)}}'

  ceph:
    memory: '{{ceph_memory|default(default_memory)}}'
    disk: '{{ceph_disk|default(default_disk)}}'
    vcpu: '{{ceph_vcpu|default(default_vcpu)}}'
    extradisks: true

  blockstorage:
    memory: '{{block_memory|default(default_memory)}}'
    disk: '{{block_disk|default(default_disk)}}'
    vcpu: '{{block_vcpu|default(default_vcpu)}}'

  objectstorage:
    memory: '{{objectstorage_memory|default(default_memory)}}'
    disk: '{{objectstorage_disk|default(default_disk)}}'
    vcpu: '{{objectstorage_vcpu|default(default_vcpu)}}'
    extradisks: true

  opstools:
    memory: '{{osptools_memory|default(default_memory)}}'
    disk: '{{osptools_disk|default(default_disk)}}'
    vcpu: '{{osptools_vcpu|default(default_vcpu)}}'

  undercloud:
    memory: '{{undercloud_memory|default(default_memory)}}'
    disk: '{{undercloud_disk|default(default_disk)}}'
    vcpu: '{{undercloud_vcpu|default(default_vcpu)}}'

node_count: 7
overcloud_nodes:
  - name: control_0
    flavor: control
    virtualbmc_port: 6230

  - name: control_1
    flavor: control
    virtualbmc_port: 6231

  - name: control_2
    flavor: control
    virtualbmc_port: 6232

  - name: compute_0
    flavor: compute
    virtualbmc_port: 6233

# - name: opstools_0
# flavor: control
# virtualbmc_port: 6234

  - name: ceph_0
    flavor: ceph
    virtualbmc_port: 6235

  - name: ceph_1
    flavor: ceph
    virtualbmc_port: 6236

  - name: ceph_2
    f...

I think the regression may be https://review.opendev.org/686193
That had changed the exit condition for the locks manager when detecting and managing colliding locks.

Fix proposed to branch: master
Review: https://review.opendev.org/688421

Change abandoned by Bogdan Dobrelya (bogdando) (<email address hidden>) on branch: master
Review: https://review.opendev.org/688421
Reason: That endless waiting seen in the timed out OVB jobs was probably caused bu that "fixed" exit condition. Apparently "keep retrying until we no longer have collisions" was not the best strategy :)

I'm not sure which should be the exit condition then. I leave that for the authors of the involved changes.

The root cause may so that is described here https://bugs.launchpad.net/tripleo/+bug/1848532

If so, then https://review.opendev.org/#/c/688660/ should close this issue as well

With applied patch deployment fails on the same step:

TASK [tripleo-container-image-prepare : Run tripleo-container-image-prepare logged to: /var/log/tripleo-container-image-prepare.log] ***
fatal: [undercloud]: FAILED! => {
    "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result",
    "changed": true
}

with following error in /var/log/tripleo-container-image-prepare.log:

2019-10-23 09:55:10,583 333650 INFO tripleo_common.image.image_export [ ] [tripleomaster/centos-binary-manila-scheduler] Layer written successfully /var/lib/image-serve/v2/tripleomaster/centos-binary-manila-scheduler/blobs/sha256:5ae90976da713f10598c8328b2dcd5ec2fb4c0dc2a3a83287add9296ce0c3a17.gz
2019-10-23 09:55:10,859 333650 INFO tripleo_common.image.image_uploader [ ] [docker.io/tripleomaster/centos-binary-manila-scheduler:current-tripleo] Completed upload for image
2019-10-23 09:55:13,284 333561 ERROR root [ ] Image prepare failed: [tripleomaster/centos-binary-nova-libvirt] Write Failure:
Traceback (most recent call last):
  File "/usr/bin/tripleo-container-image-prepare", line 138, in <module>
    lock=lock)
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/kolla_builder.py", line 235, in container_images_prepare_multi
    uploader.upload()
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 269, in upload
    uploader.run_tasks()
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 2095, in run_tasks
    for result in p.map(upload_task, self.upload_tasks):
  File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 605, in result_iterator
    yield future.result()
  File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 429, in result
    return self.__get_result()
  File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 381, in __get_result
    raise exception_type, self._exception, self._traceback
IOError: [tripleomaster/centos-binary-nova-libvirt] Write Failure:

Not sure what could be the IOError issue, undercloud storage is not full:
[stack@undercloud ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 84K 3.9G 1% /dev/shm
tmpfs 3.9G 97M 3.9G 3% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/vda 150G 44G 107G 30% /
tmpfs 799M 0 799M 0% /run/user/1000
tmpfs 500M 35M 466M 7% /var/log/heat-launcher
[stack@undercloud ~]$

It is technically possible that https://review.opendev.org/#/c/690111/ might fix that bug as well