Image for post
Image for post

Secrets and LIE-abilities: The State of Modern Secret Management (2017)

Covers KeyWhiz, Vault, Docker 1.13, DC/OS 1.8, Rancher 1.4, and Kubernetes

Jeff Nickoloff
Feb 9, 2017 · 21 min read

Current Best Practices

Evaluating solutions is easier if you have a common framework for evaluation. Here’s a quick list of centralized secret management and distribution best practices:

  • No secret should be transmitted over a network in cleartext — ever
  • All secret lifecycle and access events should be recorded in an incorruptible audit log
  • Secret distribution should be coordinated by a authoritative delegator such as a container/service scheduler or working in a close trust relationship with the scheduler
  • Operator access to secret cleartext should be limited — if not impossible without subversive efforts
  • Secret versioning or rolling should be easier to accomplish than revealing cleartext
  • All infrastructure components related to secret management and distribution should be mutually authenticated
  • Secure system configuration should be easier than advanced, and likely insecure configuration
  • The attachment of a secret to a service or container should be protected by rich (pluggable) access control mechanisms — role based access control is a plus
  • Anything that can be done to minimize the value of a secret should be done

Standalone Secret Managers

KeyWhiz by Square

KeyWhiz (Apache-2) is a secret manager in use at and produced by Square. KeyWhiz relies on an existing Public Key Infrastructure (PKI) and will use whatever database backend you wish to provide it. KeyWhiz itself is the little golden nugget at the center of a secret management system. It does one thing and well: it wraps secret data in a encryption/decryption barrier and vends that data to trusted recipients.

Image for post
Image for post
“Oh, just a PKI. That should be simple right?”
Image for post
Image for post
“I get it and I think I can work with this.”

Vault by HashiCorp

Vault is newer than KeyWhiz and a full-blown open source (MPL-2) HashiCorp product. The documentation is robust and integrations are plentiful. Vault is the current gold standard in secret management and provisioning, but it isn’t perfect either.

Image for post
Image for post
“Its just so beautiful…”
Image for post
Image for post
“I have no idea what else to ask for. This is what leading the pack looks like.”

Standalone Summary

KeyWhiz is great for its specific scope. Adopting it would be straight forward for teams already running many of its dependencies. If you’re stuck in a world full of static secrets and don’t need much in the way of audit logging it will surely fit well with your existing infrastructure.

Integrated Secret and Workload Management

Docker and SwarmKit (1.13)

I’d like to cover Docker first because it is a new kid on this block and many people may not be familiar with its secret management yet. Docker made some pretty significant shifts with their release of SwarmKit and Docker 1.12. At the time I felt that absorbing Swarm into the main project was a mixed bag. On one hand the feature set they released was clearly something that I’d rather have seen on an experimental branch. From what I could tell it also killed any momentum that composable Swarm v1 had going for it. On the other hand, based on my knowledge of the inner workings of this and other orchestration platforms and their bottlenecks I could tell that Swarm mode would scale much better and be an order of magnitude simpler to operate. It was a pivot to be sure.

Image for post
Image for post
“I can almost taste all the work I won’t have to automate myself.”
docker-machine create -d virtualbox m1
docker-machine create -d virtualbox m2
docker-machine create -d virtualbox m3
docker-machine create -d virtualbox w1
# Please pardon the sed usage - wanted you to be able to copy
# Create a cluster and put the unlock key somewhere safe
read UK <<<$(docker $(docker-machine config m1) \
swarm init \
--advertise-addr $(docker-machine ip m1) \
--autolock \
| sed -n 's/.*\(SWMKEY.*\).*/\1/p')
# Get a handle to both the manager and worker join tokens
read MJT1 <<<$(docker $(docker-machine config m1) \
swarm join-token manager | \
sed -n 's/.*\(SWMTKN.*\) .*/\1/p')
read WJT1 <<<$(docker $(docker-machine config m1) \
swarm join-token worker | \
sed -n 's/.*\(SWMTKN.*\) .*/\1/p')
# add the other two nodes as peer managers
docker $(docker-machine config m2) \
swarm join \
--token $MJT1 \
--advertise-addr $(docker-machine ip m2) \
$(docker-machine ip m1)
docker $(docker-machine config m3) \
swarm join \
--token $MJT1 \
--advertise-addr $(docker-machine ip m3) \
$(docker-machine ip m1)
# Congrats on the 3 manager Swarm cluster...
# Add your worker nodes
docker $(docker-machine config w1) \
swarm join \
--token $WJT1 \
$(docker-machine ip m1)
docker $(docker-machine config m1) node ls
# Should show 4nodes in the list with 3 managers and 3 workers
docker $(docker-machine config m1) secret ls# Create a secret named "testsecret"
echo This is the plaintext \
| docker $(docker-machine config m1) secret create \
--label justatest=1 \
testsecret -
docker $(docker-machine config m1) secret ls
# Should include your new secret in the list
# Start a service and inject a secret
docker $(docker-machine config m1) \
service create -t \
--secret testsecret \
-p 1500:1500 \
allingeek/secret-leak
# Query the service to see the leaked secret content
curl http://$(docker-machine ip m1):1500/
Image for post
Image for post
“Tastes great and less filling.”

DC/OS by Mesosphere (Enterprise subscription only)

DC/OS (Mesosphere) is a collection of open source projects (Apache-2) that have been married well by people who really know what they’re doing. Starting in 1.8 they have included secret management APIs in the Enterprise version of Marathon (the primary control-plane and resource scheduler component).

Image for post
Image for post
“Pleasant and surprising”

Rancher and Cattle (1.4.0) (Edit: Several modifications)

Rancher can be used to provision different kinds of platforms including Docker, Mesos, and Kubernetes. The default workload orchestrator — named Cattle — is also open source and included with any Rancher installation. Rancher 1.4 includes an experimental secrets management implementation. Command line and Rancher compose integration will ship with a later version. As of this writing you can manage secrets via the web user interface like you would storage, registries, etc..

Image for post
Image for post
“Stay calm, it is experimental.”
Image for post
Image for post
“Solid.”

Kubernetes (1.5.2 and earlier)

Until late 2016 I had been advocating that community users who needed integrated secret and service orchestration look into Kubernetes for a long time. I had not used it for any projects that I would consider critical and so my depth on its secrets implementation was minimal. There are many other reasons to use Kubernetes and so directing people toward an integrated solution makes sense.

  • “If multiple replicas of etcd are run, then the secrets will be shared between them. By default, etcd does not secure peer-to-peer communication with SSL/TLS, though this can be configured.”
  • It is not currently possible to control which users of a Kubernetes cluster can access a secret.
  • Currently, anyone with root on any node can read any secret from the apiserver, by impersonating the kubelet. It is a planned feature to only send secrets to nodes that actually require them, to restrict the impact of a root exploit on a single node.
Image for post
Image for post
“Move along, nothing to see here.”

Hosted Secret Solutions

PaaS companies provide a significant value add. That being said, hosted secret management is only appropriate for a specific segment of users: the risk tolerant. If hacking is more than just a financial risk to your project then never use a hosted solution.

Image for post
Image for post
“I probably won’t be hacked as bad as Yahoo!, Target, HomeDepot, DNC, Tumblr, DropBox, LinkedIn…”

Integrated Platform Summary

If you’re only interested in comparing secret management implementations that are included with service/container platforms then I think in the absence of other factors you’d be well served by both Docker, Mesosphere, and now Rancher. You can use Docker and Rancher secrets today for free and enhance Docker’s offering later with a Docker Datacenter license or you can pony up for an Enterprise DC/OS license. Just avoid any Kubernetes related secrets. Don’t use them. Anywhere.

Image for post
Image for post
“Cleartext storage… I just…”

Overall Summary

I think the most important thing to take away from this survey is that in 2017 there is some incredibly powerful secret management software that everyone is free to adopt today. Even more inspiring is that this is all open source software and everyone reading is encouraged to participate in pushing this concern forward on both a technological and prioritization front. These tools and new ones are going to keep getting better so don’t be afraid to adopt and iterate.

On Docker

Tangential thoughts and conversational notes about Docker…

Jeff Nickoloff

Written by

I'm a cofounder of Topple a technology consulting, training, and mentorship company. I'm also a Docker Captain, and a software engineer. https://gotopple.com

On Docker

On Docker

Tangential thoughts and conversational notes about Docker from my experience and research for Docker in Action.

Jeff Nickoloff

Written by

I'm a cofounder of Topple a technology consulting, training, and mentorship company. I'm also a Docker Captain, and a software engineer. https://gotopple.com

On Docker

On Docker

Tangential thoughts and conversational notes about Docker from my experience and research for Docker in Action.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store