Docker Secret Management — Best Practices

by Scott Ross

Secret Management in Docker is one of those things. Docker is just new enough that idiomatic approaches are difficult to source. There are the occasional stack overflow questions or git issues list discussing secret management, but these are not succinct. Orchestration tools are really getting a good handle on managing secrets for fleets, but there is still a gap in communication talking about what secret management is, why it’s important and things to avoid.

So that’s the challenge for this post, to tackle secret management in Docker. While we are at it, we will also talk about AWS, and how cloud services have an impact.

Okay. So what are secrets.. and what are some examples..
* database credentials
* api tokens
* ssh keys
* username and passwords
* etc.

What is buildtime and runtime, and how does it impact Docker secret management? Buildtime conceptually is Docker build , usually done on a Dockerfile. Runtime is the docker run command. I will discuss later that wrapping secrets at buildtime in a Docker file is a big “no”, but at times you may need credentials to access other parts of your build, for example package managers like ruby gems, nuget, npm or others. In those instances, you still do not want to bake credentials into your dockerfile. Instead its recommended that you leverage your build process, or use a third party tool to manage secrets.

Please, keep in mind that: build time secret != run time secret

Its also important to note why you are using docker in the first place, to create immutable infrastructure.

Best Practices:
1) Use a secrets store at run time to pass secrets. Examples of this include vault. Specifically for AWS .. you could follow something like this. There are a bunch of other utilities that could help, including Square’s Keywhiz and Sneaker (for AWS).
2) For build time secrets (accessing source control, package managers, etc al) you still should not bake credentials into your dockerfile. One possible strategy is to use your build pipeline (Jenkins pipeline, for instance) to pull all of your dependencies, and then craft your dockerfile to use those.
3) We love AWS at Cornell. We use elastic beanstalk at the Vet College for lots of deployments. Here is an example of a strategy to use KMS, AWS to secure environment variables.
4) If you have to do build time secrets, here is an example of two step building which will work.

Some processes that you may want to think about:
1) If you use environment variables, understand how they can leak and protect against that. The environment variables are visible viadocker inspect, therefore are available to any user that can run docker commands on that instance (although that user needs to be sudo on the host).
2) Volume Mounts. I am not putting this in my “don’t do list”, but this feels a lot like “rolling your own security”, and usually that turns out bad. Just be aware that straight mounting volumes doesn’t mean secrets are encrypted.

Do not do:
1) Do not put your secrets into a dockerfile, or “bake secrets” into your image.

• You do not want secrets in repositories (source control, docker hub, local dtr..)
• You do not necessarily want those secrets accessible by other users
• It breaks why you are using docker in the first place (immutability)
• Very difficult to update

2) Do not roll your own solution. There are plenty of helpful tools to manage secrets both at run time and build time. And quiet possibly, a better architecture is available.
3) Do not rely on docker squash. You still have to worry about your build cache..
4) Manually building containers. Don’t do this for other reasons.

In the conclusion, happy dockerizing! Hopefully this helped clarify some of the potentially confusing topics as it relates to secret management.