What you need to know about upcoming Docker Hub rate limiting

https://www.docker.com/blog/what-you-need-to-know-about-upcoming-docker-hub-rate-limiting/

by susam • 6 years ago

17 25 6 years ago

warpech2• 6 years ago

Is it only me or does Docker change their mind about the upcoming policy every few days, and then present it in a way that blames the users for not understanding it?

magicalhippo• 6 years ago

The Robot Chicken skit with Darth Vader and Lando comes to mind...

https://www.youtube.com/watch?v=WpE_xMRiCLE

redis_mlc2• 6 years ago

It would be great if some HNers on the anonymous and free tiers could go through the process and report on how throttling affected them in detail.

I have to admit that 100 pulls per 6 hours doesn't sound like enough for even a small team doing CI/CD.

But in the long run, it might make a lot of organizations more lean in how they do builds, which could be a good thing.

barumi6• 6 years ago

It would be great if some HNers on the anonymous and free tiers could go through the process and report on how throttling affected them in detail.

In my POV the first fundamental change that this caused was to cease considering docker hub as reliable infrastructure, and therefore started to look for solutions to reliably source container images. For instance, GitLab already provides its own container registry, and thus it makes sense to simply source images from them than from tertiary sources like Docker Hub.

Another impact that this has is that I'm looking into the need to operate a custom container registry for each Kubernetes cluster. It wouldn't be nice if an application running on Kubernetes or Docket Swarm risked downtime just because you happened to hit one of Docker Hub's rate limits.

redis_mlc2• 6 years ago

Well, the throttling seems to be on anonymous and free docker hub accounts for now. So you do have the option of paid tiers.

I do recall reading about the free pulls a year ago and thinking, "Wow, that's awfully generous!" :)

chrisandchris• 6 years ago

Exactly this. It‘s so easy to setup your registry using GitLab or then Docker Registry, Portainer and Traefik. No need to rely on external providers for a small team.

Sahbak• 6 years ago

Unless I'm mistaken, 100 pulls per 6 hours per IP is a lot - You would/should be caching most of the images from dockerhub anyways, so subsequential builds shouldn't count against your limit.

szszrk• 6 years ago

Internal repo that does caching of images is rather obvious and a good practice, but it still doesn't mean that 100 pulls are that many.

Maybe we understand something different by a "small" team, but IMHO a smaller team uses more ready public images, to save time and costs. Also even not that big project on Kubernetes that includes internal software will also have a lot of other software.

Let's speculate: I had a look at some teams I know and how they work (or they would like to work soon) and it's easily 15-30 images in a CI/CD pipeline. Plus tools used to develop, but not exactly released in a build (source code management, PM, communication etc). All teams I know have 2-3 cycles developed at all times (next release, hotfix, future release after the next one). This means a lot of these tools are multiplied by 2-3 (upgrading tools as well, so multiple versions of same tool needed). If you have a few clusters that have different requirements it can also be multiplied by 2 or 3.

I would not dare to think what would happen if someone used "latest" tag in their daily work, or organized a docker/kubernetes workshop for new people :)

barumi6• 6 years ago

Unless I'm mistaken, 100 pulls per 6 hours per IP is a lot

It really depends on your usecase. I mean, each CICD stage can involve building a brand new image and running a couple of them, sometimes more. 100 pulls per 6 hours corresponds to a constant rate of 3 min between pulls, which would require between 15 to 20 minutes for each commit.

Anyway, the primary practical implications of this sort of change is a) rate limiting suddenly becomes a requirement, b) we can no longer count on Docker Hub to provide a reliable service, thus we plan accordingly.

robrtsql• 6 years ago

At one point, Docker's announcement said that all 'docker pull' requests would count towards the rate limit, even if you had every layer cached and no layers ended up being downloaded.

I may be wrong but I interpreted that to mean that even caching won't prevent you from hitting the limit.

franga20006• 6 years ago

If your CI is doing more than 100 pulls and you have a "small team" then you must be doing something wrong. Either you have way too many project for a small team to work on, or your CI configuration is incredibly wasteful.

Just a few years back I remember even the smallest Java shop would have a local "caching/forwarding" Maven repository and the upstream repos were only considered a fallback.

Has this mentality not continued into the Docker era? It seems pretty easy to set up [0] and has all kinds of advantages even without the upcoming limit.

[0] https://docs.docker.com/registry/recipes/mirror/

barumi6• 6 years ago

If your CI is doing more than 100

A limit of 100 pulls per 6 hours averages to a pull each 3 minutes. Each stage of a CICD pipeline uses more than 1 container image, and it's customary to have CICD stages build and pull from previous stages, such as integration tests,performance tests, canary tests, etc.

I would dare say that you're doing something terribly wrong if you don't need to pull images often, because that largely tells you're doing barely any tests or deployments.

jacques_chester2• 6 years ago

If your CI is doing more than 100 pulls and you have a "small team" then you must be doing something wrong.

Most CI systems poll to see if there are new versions of an image. If this is done with a GET request (to get the manifest as a first step towards pulling layers), it counts towards the rate limiting.

Has this mentality not continued into the Docker era? It seems pretty easy to set up [0] and has all kinds of advantages even without the upcoming limit.

Maven separates the domain of a Maven repository from the coordinates of packages. Docker does not. Dockerhub's domain is inferred into any shorthand reference like `foo/bar`, meaning that it will ignore a pull-through cache.

You can ostensibly configure a docker daemon to treat another registry as a trusted mirror, but it's harder than it looks in practice, because on most production systems that won't (and shouldn't) be enabled.

physicsguy• 6 years ago

The biggest issue is that if you run the command 'docker pull X', it counts as a pull even if you already have the up-to-date image, and so it doesn't do a download.

warpech2• 6 years ago

If I understand correctly, there might be an agreement between Docker and GitHub, by which the use of public Docker Hub images won't be rate limited in GitHub Actions: https://github.com/actions/virtual-environments/issues/1445#...

What you need to know about upcoming Docker Hub rate limiting

Related Stories