Securely build small python docker image from private git repos

Before I start I wanna give credit to Tõnis Tiigi for writing this excellent blog post, some of the excerpts here are his words.

One of the complexities of building Docker images for private repository was always the challenge of passing credentials for private Git repositories to the Docker build process using the Dockerfile.

There really wasn’t a very good solution for it and all other solutions seemed somewhat hacky:

• Using Environment variables, Args:

ARG SSH_PRIVATE_KEY
RUN mkdir /root/.ssh/
RUN echo "${SSH_PRIVATE_KEY}" > /root/.ssh/id_rsa
# [...]
RUN rm /root/.ssh/id_rsa

Removing the SSH key is pointless since its still in one of the layers and would still remain in the metadata of the image.

• The introduced --squash flag in Docker 1.13 will remove files which are not present anymore, and reduce multiple layers to a single one, however, a single layer is slower to download and re-builds suffer from lost layers that would have been used in caching.
• Build your code and dependencies into wheels as a step before the build, and in the build add that to the image.
  Before the days of MultiStage build, I wrote a native python solution to package your code and dependencies with setuptools.

Docker Multi-Stage, a step forward

A new way of building Docker images without the need for extra build scripts was added in Docker 17.05

FROM golang:1.7.3
WORKDIR /go/src/github.com/alexellis/href-counter/
RUN go get -d -v golang.org/x/net/html  
COPY app.go .
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o app .

FROM alpine:latest  
RUN apk --no-cache add ca-certificates
WORKDIR /root/
COPY --from=0 /go/src/github.com/alexellis/href-counter/app .
CMD ["./app"]

So we could use that to securely add a throwaway stage that would be used just for the sake of passing a secure secret or SSH key

# ====================
# STAGE 1: base
# ====================
FROM python:2.7.15-alpine AS base
RUN apk update && \
    apk add git && \
    apk add --virtual build-deps gcc python-dev musl-dev && \
    apk add postgresql-dev && \
    rm -rf /var/cache/apk/*
RUN mkdir -p /code
WORKDIR /code
# ====================
# STAGE 2: unsecure
# ====================
FROM base AS unsecure
# Get the sensitive credentials
ARG GIT_USERNAME
ARG GIT_PASSWORD
# Set up git credentials
RUN git config --global credential.helper 'store --file /git-credentials'
RUN echo "@github.com">https://${GIT_USERNAME}:${GIT_PASSWORD}@github.com" > /git-credentials
# Install pip requirements
COPY requirements.txt .
RUN pip install --no-cache -r requirements.txt
# ====================
# STAGE 3: secure
# ====================
FROM base AS secure
COPY . /code
COPY --from=unsecure /usr/local/lib/python2.7/site-packages /usr/local/lib/python2.7/site-packages
COPY --from=unsecure /usr/local/bin /usr/local/bin

The final image would not include the unsecure step (only the first and last step).

Still, it’s close but not really there since these layers are on your computer somewhere…

Enter Docker SSH Forwarding and Secrets Mounts

The build command in Docker 18.09 comes with a lot of new updates. Most importantly it can now use a completely new backend implementation that is provided by the Moby BuildKit project. BuildKit backend comes with a bunch of new features, one of them being build secrets support in Dockerfiles.

With the new Docker build kit you can use one of the 2 options:

• SSH:
  You can use the--ssh flag to forward your existing SSH agent key to the builder. Instead of transferring the key data, docker will just notify the builder that such capability is available. Now when the builder needs access to a remote server through SSH, it will dial back to the client and ask it to sign the specific request needed for this connection. The key itself never leaves the client, and as soon as the command that requested the access has completed there is no information on the builder side to reestablish that remote connection later.
• Secrets:
  Provides a mount option during the build at /var/run/secrets available only for the command that used it and is not included in the created layer.

The first thing to do once your docker version is 18.09 and up, is to enable the build with an environment variable DOCKER_BUILDKIT=1before running docker build.

export DOCKER_BUILDKIT=1

more about build kit here

Add to the top of your dockerfile the new syntax:

# syntax=docker/dockerfile:1.0.0-experimental

For this post, I'm only going to concentrate on the SSH option.

Let’s write our dockerfile, our docker file will have the secure option for the SSH mount, and also a way to package the python dependencies into wheels for installing in the final stage.

In the Dockerfile above we create wheels by using the --ssh flag and we slim down the image to about 100MB (your mileage may vary depending on the amount and size of python packages you install)

Note the following lines on stage 1:

RUN --mount=type=ssh,id=github_ssh_key pip wheel \
--no-cache \
--requirement requirements.txt \
--wheel-dir=/app/wheels

we are giving a key name github_ssh_key so we can use it when we invoke docker build like so:

docker build --ssh github_ssh_key=/path/to/.ssh/git_ssh_id_rsa .

only the agent connection is shared with that command, and not the actual content of the private key.
no other commands/steps in the Dockerfile will have access to it.

This concludes the overview and short tutorial for making a secure docker image using the new --ssh flag and new docker 18.09 syntax. This has helped me to build private images with ease that can be invoked on any platform that runs the new Docker.

Did you know you can clap more than once? hold for the applause effect :)