Unable to push to swift storage backend

I can reproduce the error manually by deleting a segment from a layer:

'swift delete docker-registry segments/2f6/46f636b65722f7......'

Or all of them like this):

'swift list docker-registry |grep segments |xargs swift delete docker-registry'

The following seems to recover things consistently:

REGISTRY=$(docker ps |awk '{print $1}'|tail -n 1)
CONTAINER=docker-registry
FILES=$(docker logs $REGISTRY | jq -r -R 'fromjson? |.["err.detail"] | match("swift: Timeout expired while waiting for segments of (.+sha256.+) to show up"; "g") | .captures[0] .string' |sort |uniq)
for f in $FILES; do
        swift stat --lh $CONTAINER "files$f" 2>&1 |grep -q "Content Length: 0"
        if [ $? -eq 0 ]; then
                echo "Removing $f"
                swift delete $CONTAINER "files$f" > /dev/null 2>&1
        fi
done

Then push again and things will work.

This workaround isn't feasible for us, as there's no way of running it at the point of "docker push", as it requires access to the docker registry logs and swift write access.

However, we suspect, based on reading some of the upstream issues on GH and where we've seen this issue happening, that this may have something to do with the proxying setup. We're going to try a few different permutations and will update this bug report once we've done that and confirmed (or otherwise).

Here's some more data to add to this issue, and hopefully narrow things down:

These two configurations will reliably trigger the issue about 15-20% of the time. During testing, I almost thought the Haproxy & docker only setup with balance source to pin sessions was working but I hit the bug there too.

1. 2x Apache front-ends, 2x Haproxy mid-ends, 2x docker-registry backends.
2. 2x Haproxy front-ends, 2x docker-registry backends

Interestingly, stopping one docker-registry backend (systemctl stop docker) and then pushing doesn't seem to trigger the bug.

To that end, manually generating an Haproxy configuration and marking one server as a 'backup' seems to also avoid the issue.

Here is an Haproxy configuration that can be used to reproduce the issue given the above. I created this configuration by hand. Doing it with the juju services config option for Haproxy seems difficult during testing given the relations and need to bind on port 80 & 443 in the same frontend.

---------------------------------------------------
global
    log /dev/log local0
    log /dev/log local1 notice
    maxconn 4096
    user haproxy
    group haproxy
    spread-checks 0
    tune.ssl.default-dh-param 1024
    ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:!DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:!DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:!CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA

defaults
    log global
    mode http
    option httplog
    option dontlognull
    retries 3
    timeout queue 20000
    timeout client 50000
    timeout connect 5000
    timeout server 50000

frontend haproxy-1-443
    bind 0.0.0.0:80
    bind 0.0.0.0:443 ssl crt /var/lib/haproxy/default.pem no-sslv3
    mode http
    option forwardfor
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    redirect scheme https if !{ ssl_fc }
    default_backend docker-registry_be

backend docker-registry_be
    mode http
    balance source
    hash-type consistent
    redirect scheme https if !{ ssl_fc }
    option httpchk GET / HTTP/1.0
    server docker-registry-2-5000 10.55.32.146:5000 check inter 2000 rise 2 fall 5 maxconn 4096
    server docker-registry-3-5000 10.55.32.153:5000 check inter 2000 rise 2 fall 5 maxconn 4096
---------------------------------------------------

1. Steps to reproduce:
juju deploy haproxy
juju add-unit haproxy
juju deploy cs:~containers/docker-registry-25
juju add-unit docker-registry
juju add-relation haproxy docker-registry

2. Add DNS entries for the haproxy units, e.g. I used registry.jamon.ca.

3. Generate a self-signed TLS key/cert pair the registry and configure Haproxy to use them:
juj...

So this looks to me like we need to update the docker-registry relation to HAProxy so that it configures things in active/passive rather than active/active mode - i.e. any HAProxy sitting in front of docker-registry should only send traffic to one of the docker-registry backend unless that backend is down. This will mean we're not getting the load-spreading benefits of multiple docker-registry units, but should solve this issue.

Marking this as critical as it currently affects any deployment with more than one docker-registry unit and HAProxy infront of it.

Fixed with https://github.com/CanonicalLtd/docker-registry-charm/pull/23, currently in the beta channel as cs:~containers/docker-registry-65

This was a relatively simple fix that only configures the haproxy for the docker-registry leader. As Tom mentioned, we'll lose load balancing capabilities, but gain a consistent proxied registry with hot-spare capabilities.

This issue is still causing us problems in production. (see https://rt.admin.canonical.com/Ticket/Display.html?id=115963)

Martin, we confirmed the original fix didn't always work between docker-registry-65 and -87. The issue was that registry leadership did not enforce anything about the order of multiple registries, meaning multiple proxies fronting multiple registries could disagree about the order of registries to contact, which led to multiple proxies talking to different backend registries, which would re-trigger this bug.

This should be fixed for good in docker-registry >= 87, as discussed in this bug:

https://bugs.launchpad.net/layer-docker-registry/+bug/1815459

And this PR:

https://github.com/CanonicalLtd/docker-registry-charm/pull/26

Please upgrade to the latest stable rev (anything >= 87 should be fine).

Hello! Unfortunately, this bug is still showing up.

I've done a fair amount of research and am fairly certain that the issue we are encountering is specifically this one:

https://github.com/docker/distribution/issues/2188

What appears to be the actual bug is that the swift backend does a swift.move, and then removes the temporary upload files. This, combined with swift's internal handling of move can result in missing segments and an essentially corrupt object which has to be removed manually in order to un-stick the related image.

A workaround is to trigger a complete rebuild of the image, so no layers are reused, essentially stepping around the broken object.

An excellent explanation of the actual issue can be found in this docker PR:

https://github.com/docker/distribution/pull/2878

Applying the patch in the PR may be the most direct path to a solution to this problem in the short term.

There is another backend driver also, but it is not officially supported, and the PR above seems to be a less invasive solution.

I don't see docker-charm changes required here. If I'm reading comment #8 correctly, the deployment simply needs to be configured with jetpackdanger's image:

juju config docker-registry registry-image='jetpackdanger/registry:pr2878'

Let me know if there's more todo for the charm. Otherwise, I think we workaround by setting a custom registry image until upstream accepts:

https://github.com/docker/distribution/pull/2878

Per irc discussion, the workaround from comment 9 makes this bug less critical. Long term, we'll either need upstream to merge pr2878, or adjust the default registry-image config in this charm to ensure it works well for swift backends.

I'm not comfortable changing the default config today as the custom image hasn't been tested against non-swift backends. Todo here is to beef up this charm's tests to gain confidence that our defaults are safe (and fix the original swift issue).