OCI: Hitting Dockerhub's pull rate-limit when doing mass-rebuilds of OCIs

I am not sure what we can do from the Launchpad side to solve this problem.

Could you elaborate?

A solution on the ROCKs end would be using an internal registry which could be used by launchpad (and possibly only by launchpad) to fetch the images from it instead of using third party registries (the built images would need to be pushed there as well).

From a launchpad perspective - until there is an internal registry available - we could authenticate to ECR prior the build step and perform builds from the images in there instead of using DockerHub. With a 10 image pulls/second rate we could perform 2 simultaneous builds at any given time. While we may still experience breakage with this approach (when more than 2 builds try to pull images for all architectures within the same second), a re-trigger would be enough to perform the next build.

With dockerhub's current 100 pulls/6 hour rates, we can only build 25 images every 6 hours, which is no longer enough for mass rebuilds on all the ROCKs we maintain.

So, should this bug be a request for ECR authenticated pulls instead?

On a side note, this issue also blocks us from using automated daily re-builds of the OCI images. Moving to pulling from ECR would not help if all the builds would be triggered at the exact same time everyday though.

[1] https://docs.aws.amazon.com/AmazonECR/latest/public/public-service-quotas.html

We also have Docker Team Licenses for accounts part of the Ubuntu organisation. Logging in against Docker Hub with the service account credentials used for the push rule should provide a rate of 5,000 pulls/24 hours (instead of the unauthenticated 100 pulls/6 hour) [1].

However, the best solution would be the use of an internal registry, could be private to Launchpad at first indeed. This used to be in progress, what's the status on this?

[1] https://www.docker.com/pricing/resource-consumption-updates#:~:text=Daily%20pulls%20for%20an%20individual,Docker%20hub%20per%2024%20hours.

The Harbor deployment is blocked partly on having run into Juju bugs such as https://bugs.launchpad.net/juju/+bug/1948695, but mainly on time, since we've had multiple other drop-everything-priority projects come up for us this cycle and that makes everything difficult.

It might be technically feasible (and even make sense more generally, e.g. for private builds) to dispatch the registry credentials that we'd use for pushing a given recipe to builds of that recipe. However, that would break the security policy we have for registry credentials (because anyone with access to perform builds of a recipe would be able to exfiltrate its registry credentials via a suitably-crafted build), so it's not clear to me that we could do that - we'd need to discuss whether it would be viable.

(Though possibly we could do `docker login` and ensure that `docker build` doesn't see the raw credentials. I don't know how `docker login` stores credentials.)

We discussed this in our infrastructure call today. Dispatching push-capable credentials to builders is a non-starter for security reasons, but we could dispatch pull-only credentials if they existed (it would be some work to set up and we'd need to figure out exactly where to put them in the data model, but certainly less work than finishing the Harbor deployment). Does Docker Hub have any way to get pull-only credentials that would have the right sort of effects on rate limiting?

When requesting a token from the registry API, we can set the scope in the request:

https://docs.docker.com/registry/spec/auth/token/#requesting-a-token

If we are using a permanent token instead, docker hub does provide scoped tokens for team accounts as per https://docs.docker.com/docker-hub/access-tokens/

[Expired for Launchpad itself because there has been no activity for 60 days.]