Don’t Embed Configuration or Secrets in Docker Images

6 min readSep 20, 2017

tldr;

Never embed configuration or secrets into a Docker image. Instead, when building a Docker image, expect that any required runtime configuration and secrets will be supplied to the container using the orchestration runtime (Kubernetes Secrets, Docker Secrets), an external tool (Hashicorp Vault), or environment variables (for non-sensitive data). Sane configuration defaults are ok (even recommended). Be careful to not inadvertently include secrets in hidden layers of an image.

Overview

Container images should be both reusable and secure. When an image contains embedded configuration or secrets it violates this rule and is either not reusable or is insecure. These values don’t belong in the image, they belong only in the running container. When credentials are left in a Docker image, there are security implications and it could create outages.

Running a Docker container in production should be the assembly of an image with various configuration and secrets. It doesn’t matter if you are running this container in Docker, Kubernetes, Swarm or another orchestration layer, the orchestration platform should be responsible for creating containers by assembling these parts into a running container.

Types of Docker Images

Component Images

A Docker image is often a reusable component that is shared with different infrastructure and designed to run in various environments. An example of this type of image is Postgres, Redis or NSQ. These components often require some credentials to bootstrap. The credentials should be supplied by the container, not built into the image.

A good example of how this is done is the official MySQL image. If you run the MySQL Docker image without providing initial values for some config, the container exits:

$ docker run mysql
error: database is uninitialized and password option is not specified
  You need to specify one of MYSQL_ROOT_PASSWORD, MYSQL_ALLOW_EMPTY_PASSWORD and MYSQL_RANDOM_ROOT_PASSWORD

The MySQL image does a great job of showing what value was missing and giving you a suggestion. Trying again:

$ docker run -e MYSQL_ROOT_PASSWORD=asecret mysql
Initializing database
2017-09-19T16:10:15.862081Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2017-09-19T16:10:16.222339Z 0 [Warning] InnoDB: New log files created, LSN=45790
2017-09-19T16:10:16.286131Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2017-09-19T16:10:16.293280Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 0661a3dd-9d55-11e7-9e12-0242ac110002.

<...snip...>

This time, with some config options provided, it works. The MySQL image is reusable and secure because it can be configured to work in most environments, and it doesn’t have a password embedded into the image.

Application Images

An image can contain an application instead of a component. An example of an application image is a static site or an API. These are images that are designed to serve requests from users, instead of other components. Often, these application images need secrets such as API keys or database credentials, or these need environment-specific data such as other endpoints to connect to. Storing these secrets, other than dev-mode defaults, in the image itself is a bad practice and should be avoided. Secrets should always be provided by the container, not built into the image.

A publicly available front-end image to examine is Kibana. Kibana is often used as a UI in front of Elasticsearch (and Logstash). A Kibana container requires some configuration — at the very least it must be able to find an Elasticsearch server to connect to. The elastic.co team decided that a sane default was to connect to a host named “elasticsearch” on the default port numner (9200), because that would work well in a docker-compose (dev) environment. But hopefully nobody is running an Elasticsearch cluster in production using the username “elastic” and the password “changeme”!

Hidden Layers

Finally, it’s important to not include secrets in hidden layers of an image. This could happen when creating an image, if the image is assembled incorrectly. When you remove something in a Dockerfile (a secret, a file, source code, anything), it doesn’t always mean that this secret is not being included in the final image. It may be there in a parent layer, and still accessible. All base layers are delivered when a user runs a docker pull command.

Take the following Dockerfile for example:

FROM golang:1.9ADD ./ /go/src/github.com/my/project
WORKDIR /go/src/github.com/my/project
RUN go build -o /usr/local/bin/my-project
RUN rm -rf /go/src/github.com/my/projectENTRYPOINT [/usr/local/bin/my-project]

The developer of this Dockerfile is expecting to create an ad-hoc build environment in the Docker build process. When built, it’s expected to add the source code to the image, build the binary and then remove the source code. And when running the produced image, this appears to be what happens. The problem is that each line in a Dockerfile produces a new image which is shipped whenever the final image is pulled. The process that creates this Dockerfile also leaks the source code to anyone who pulls this image.

Why This Matters

Docker images should be functional, consistent, reusable (sharable and discoverable), small, secure, work in enterprise environments and debuggable.

Reusable

Application portability is one of the biggest benefits of deploying containers. When your application components are portable, it’s easier to deploy immutable infrastructure across various cloud providers and environments. Containers with built-in configuration and/or secrets are not portable, they would require a different image for each environment.

Secure

Images with embedded secrets are not secure. Anyone with access to the image will be able to pull and examine secrets, even if they are in masked layers. This is unaudited viewing of secrets, and should be avoided.

Enterprise Compatible

Great container images contain all of the best practices when running a component. Removing config from a container image will allow enterprise users to benefit from the best practices that are embodied in the container and still be compatible with their audit and security policies and their workflows. Built in configs will require that enterprise uses create a private image and lost much of the benefits of the best practices already in the original image.

Debuggable

Images with embedded configuration are a lot more difficult to debug. If an image has embedded config, it’s often more difficult to start a debuggable container from the image with a simple docker run command.

Recommendation

There are many ways that a container can be provided secrets and configuration at runtime:

Environment variables
Config files
etcd or other config management service
Docker secrets
External secret management solutions

Expect that, even if you are building an image that will only be used internally and only on a single server, that the container will be created with a way to access secrets. Never provide default secrets in your images. Instead of hard coding and packaging secrets in the image, supply them at runtime the container.