Docker Hub

Elastic CI Stack for AWS, authenticating with a paid Docker Hub account

If you're using the Elastic CI Stack for AWS, you can authenticate with Docker Hub by adding two environment variables to your secrets bucket and accessing them from your build.

Add your Docker Hub credentials to one of the following two environment hooks, which are downloaded at the start of each job:

• /env - An agent environment hook, run for every job the agent runs
• /{pipeline-slug}/env - An agent environment hook, specific to a pipeline

Either one of these could be configured with Docker Hub credentials to ensure Docker Hub requests are authenticated:

#!/bin/bash

DOCKER_LOGIN_USER="the-user-name"
DOCKER_LOGIN_PASSWORD="the-password"

Other Buildkite agents authenticating with a paid Docker Hub account

All agents check the local filesystem for hook scripts to execute during a job.

A pre-command hook script like this is one option for authenticating with Docker Hub, and can be configured to fetch credentials from the system you use to store them in:

#!/bin/bash

echo "~~~ Logging into Docker Hub"
docker login --username "the-user-name" --password-stdin << "the-password"

Using docker-login plugin to authenticate with a paid Docker Hub account

The docker-login plugin can perform the authentication in only the steps that need it.

Start by setting the password in an agent environment hook:

#!/bin/bash

DOCKER_HUB_PASSWORD="the-password"

Then add the plugin to pipeline YAML steps that need it:

    steps:
      - command: ./run_build.sh
        plugins:
          - docker-login#v2.0.1:
              username: the-user-name
              password-env: DOCKER_HUB_PASSWORD

Mirroring Docker images into Google Container Registry

If you're using Google Cloud, Google has some documentation on mirroring Docker Hub images to a private Google Container Registry (GCR).

This approach requires:

1. A regular process (for example, nightly) that mirrors the Docker images you need
2. Updating all pipelines to use the mirrored GCR image (for example, from nginx:1.14.2 to gcr.io/<GCR_PROJECT>/nginx:1.14.2)

Mirroring Docker images into AWS Elastic Container Registry

AWS doesn't have specific documentation, however their advice on dealing with Docker Hub rate limits suggests mirroring public images into AWS Elastic Container Registry (ECR). A similar solution to the one proposed by Google Cloud Platform (GCP):

1. A regular process (for example, nightly) that mirrors the Docker images you need
2. Updating all pipelines to use the mirrored ECR image instead of the original one

Configuring the docker daemon to use the GCR mirror of popular Docker Hub images

GCP host a mirror of popular Docker Hub images in the mirror.gcr.io registry.

For agents running on GCP, it's possible to configure docker to try the mirror first, and transparently fall back to the public Docker Hub registry when the mirror doesn't have an image. Google have documented how to set this up.

This will avoid the rate limits for many, but Google don't guarantee which images will be on the mirror, so depending on the specific images in use you may continue to hit Docker Hub rate limits.

Running a read-through caching registry

There are two popular options for running a private caching Docker registry, where requests for missing images result in the image being fetched from an origin registry (like Docker Hub).

• https://docs.docker.com/registry/recipes/mirror/
• https://github.com/rpardini/docker-registry-proxy

Once the caching registry is operating, pipelines can be updated to use images from that registry (for example, from nginx:1.14.2 to example.com/nginx:1.14.2) and new images will be transparently fetched from Docker Hub.