Fixing Docker Hub Rate Limiting Errors in CI/CD Pipelines

Add Docker Hub Credentials and Prosper. AWS CodeBuild Instructions Included.

Edoardo Nosotti
Oct 2, 2020 · 7 min read
Photo by Matthew Cooksey on Unsplash

In an effort to leave its troubled recent past behind, Docker has significantly revamped its business model over the course of this (very troubled indeed) year. As a part of this renovation, back in August the company announced big changes to the Docker Hub service too. Effective November 1, 2020:

  • Stale (inactive for ≥6 months) images on the Docker Hub will be purged
  • Pulls will be rate limited for anonymous users and free accounts

We should all know by now that there are no such things as unlimited services, capacity or vacations in the real world. Even huge companies faltered and crumbled under the burden of “unlimited” paid services. How is Docker supposed to pay for 4.5 petabytes of free storage for inactive images?

So I totally understand Docker’s change of course. Also, dealing with clear and predictable usage limits is easier than handling uncertainty.

In spite of the assurances given by the Docker team, many CI/CD pipelines will be impacted, especially if built on SaaS/PaaS solutions. Also, it looks like some limits are being enforced well before the November 1 deadline. It’s September 30th at the time of writing and today some CodeBuild(s) in my CodePipeline(s) failed with this error:

Note the error description in the last line. The Docker Hub itself is suggesting that rate limiting is already enforced.

UPDATE — In this post, the Docker team has further clarified the impact that a small fraction of “heavy & anonymous users” had on their infrastructure. This seems to confirm my considerations in the next paragraph.

Understanding the issue

For this project, I am using a standard Ubuntu 18.04 base image to build my application. Since I am pulling a public, official, reference image from the Docker Hub authentication is not required. As an anonymous user I should be granted 100 pulls every 6 hours*, more than enough to build one application in the pipeline.

The pipeline for my application though is built on AWS CodePipeline. Many other third party pipelines are built on such service and share the same pool of hosts and IP addresses. This likely leads to limit exhaustion as reported in this CircleCI thread and impacts many other managed CI/CD services.

(*) limit for anonymous pulls from the Docker Hub at the time of writing

Designing the solution

The simple way out of this is to have individual limits enforced for your pipelines. This means logging in to the Docker Hub before pulling. Just signing up for a free account will grant you 200 pulls in 6 hours*. Or you can get “unlimited” pulls for $5/month*.

(*) plans available at the time of writing

Using an account to authenticate requests to the Docker Hub is as simple as adding a $ docker login command to your build scripts. Protecting your credentials is the tricky, but absolutely necessary part.

First, the credentials should be applied at runtime from a secure storage. Never hardcode them into the build scripts. Those scripts are usually committed to a repository. Depending on the CI/CD service you are using, better options will be available. I am using AWS CodeBuild and it can set environment variables at runtime from the Parameter Store or the Secrets Manager. Travis CI lets you encrypt environment variables. CircleCI uses HashiCorp Vault to secure the environment variables. Jenkins has built-in support for credentials. Search through the documentation of the CI/CD service or automation tool of your choice to find the best way to safely inject the credentials at build time.

Second, passing your password in the command line with the --password flag is definitely a bad idea. It could be shown in the output or logged somewhere. Some CI/CD services are smart enough to filter out secrets from output and logs, but don’t just take it for granted. $ docker login can read the password from STDIN, but since the pipelines are automated you cannot type the password in yourself. Passing the credentials via environment variables, you can safely pipe the password into the command with the --password-stdin flag. Assuming that the DOCKERHUB_USERNAME and DOCKERHUB_PASSWORD environment variables have been set at runtime, add this command before any "pull” or $ docker build:

Implementing the solution in AWS CodeBuild

The solution shown above is easy to implement in AWS CodeBuild. For this exercise, I will be using the Parameter Store as a source for the credentials. Using the Secrets Manager is pretty much the same. The Parameter Store though has a generous free tier and supports encryption via KMS as well.

  • Go to the Docker Hub and sign up for a new account if you don’t have one.
  • Go to AWS KMS and create a new key. You can use an existing key if you already have one. Just don’t use the same key for everything in your account, all right? ;)
    Create a symmetric KMS key as shown in the screenshots below, do not select any user accounts at steps 3 and 4.
Create “Symmetric” key from KMS material
Name your key
This is how your generated policy should look like at step 5
  • Go to the Systems Manager Parameter Store and create 2 new parameters, one for the Docker username and one for the password. Use paths in names and choose a proper naming convention, such as: /{APPLICATION_NAME}/{ENVIRONMENT}/{PARAMETER_NAME}
    The Standard Tier should be enough for the credentials, but remember to choose the SecureString Type as shown below so they will be encrypted. You will also need to select the KMS key to encrypt the values with, choose the KMS you have created for this job.
How to set options for the Docker username and password parameters
  • Edit your CodeBuild project, go to the Environment panel and add 2 new environment variables for the username and the password. Ensure that the “Parameter” Type is selected and set the Value fields to the parameters’ path/names as you have entered them in the Parameter Store. Also, note the Service Role name, you will need to edit this later in IAM.
How to make CodeBuild set environment variables using values from the Parameter Store
  • Go to IAM / Roles, search for the Service Role attached to the CodeBuild project and edit it. CodeBuild must be granted read access to the username and password parameters in the Parameter Store and usage of the KMS key for decryption. Either edit its attached policy or attach a new policy, then add the following statements:

Replace {AWS_REGION} with the actual region name you have deployed your resources into, such as us-east-1.
Replace {YOUR_AWS_ACCOUNT_ID} with your actual account ID.
Note the /parameter/my-application/dockerhub/* path. This is to grant CodeBuild read access only to the Docker username and password.
/parameter is a default prefix and must be prepended to the parameters’ names.
/my-application/dockerhub is a prefix-path we have set in the parameters’ names when we have created them in the Parameters Store.
/* is a wildcard to allow access to all parameters under the
/parameter/my-application/dockerhub path.
Replace {YOUR_KMS_KEY_ID_HERE} with your actual KMS key ARN, you can get it from the KMS dashboard.

$DOCKERHUB_USERNAME and $DOCKERHUB_PASSWORD are the same names of the environment variables we have set in the CodeBuild project Environment configuration, prepended with $ because they are used in a shell command.

  • Commit your updated buildspec.yml and start a new build job in CodeBuild. Tail the logs from the console to ensure the login to the Docker Hub is successful.

Enjoy!

Further reading

The tutorial above shows how to authenticate requests to the Docker Hub to pull official images from their registries and use them to build your own Docker images in CodeBuild. Such build processes run within a Docker container themselves, called runtime environment. If you want to use your own private base images as a runtime environment in CodeBuild, follow this official AWS tutorial.

The official AWS tutorial on building Docker images with CodeBuild provides useful insights too.

RockedScience

Tutorials, tips and fast news on Cloud, DevOps and Code

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store