Secrets management guide — approaches, open source tools, commercial products, challenges and questions

21 min readOct 7, 2018

Secrets management is a hard problem. Many different approaches and tools are out there as well as new innovations in the space. Because we are deeply focused on this emerging space at CryptoMove, we put this post together as a resource to anybody who is thinking about or trying to learn more about secrets management. If this is an area of interest for you, we are running a private beta for CryptoMove’s Tholos Key Vault now—and we’re hiring!

Outline

I. Default secrets management approaches without tooling
II. Why secrets management matters
III. Options, tools, and commercial solutions for secrets management
IV. Key objectives for secrets management (pun intended)
V. Challenges around secrets management

I. Default secrets management approaches

Usually secrets management defaults to whatever is easiest and fastest. When devs are working together on an image in AWS, GCP, Azure or some other cloud native environment, it’s not uncommon to see .PEM files, SSH keys, and other configuration secrets passed around in cleartext and across a variety of communications channels. Some of the most popular “anti-patterns”:

Email
Slack
GitHub
Hard-coding
Clear text config files
Unencrypted laptop filesystems

There are plenty of examples of situations where the default “fast & easy” approach to secrets management and sharing have led to issues with devops & cloud transformations.

Paramount Pictures, Comedy Central, MTV and hundreds more exposed in Viacom AWS leak | SC Media

A mishandling of Viacom's master AWS key has left the credentials of hundreds of digital properties, including Comedy…

www.scmagazine.com

Accenture left a huge trove of sensitive data on exposed servers

Technology and cloud giant Accenture has confirmed it inadvertently left a massive store of private data across four…

www.zdnet.com

Developers keep leaving secret keys to corporate data out in the open for anyone to take

The hackers who stole data on 50,000 Uber drivers in 2014 didn't have to do much hacking at all. They got into the…

qz.com

II. Why secrets management matters

Due to changes in infrastructure and software development processes, secrets are proliferating widely. Here are a few ways enterprise transformations are affecting secrets management at scale:

Cloud native development & multi-cloud infrastructure leads to secrets proliferation. As teams develop cloud-native applications, secrets to services for storage, compute, analytics, logging, and dozens of other services become important to share and manage. AWS has close to, if not more than, 100 services alone, all that need to be mediated with secrets, including API keys, SSH keys, tokens, certificates, and configuration.
Machine identity and machine-to-machine communication matters as much or more than user identity. In traditional enterprise architecture, human users were the key. Human identities were important to access documents, spreadsheets, email, and other tools. Modern enterprises, however, often have tens of thousands of machine-based identities that need to be managed and mediated via tokens, API keys, certificates, and other secrets. Automation and AI/ML will continue to shift landscapes from human user identity to machine-based identity.
DevOps processes & microservices based architecture also leads to secrets proliferation. Teams undergoing DevOps transformations move fast and manage many different infrastructure environments and services for development, testing, integration, and deployment. Secrets management for DevOps environments is vital as part of the secure software development lifecycle.
AI & data analytics leads to lots of secrets to manage for these pipelines as well.
IoT, robotics, and embedded device proliferation leads to secrets proliferation, due to the need to have encryption and certificates for each IoT endpoint.
Blockchain projects in the enterprise also lead to more private keys than typically are used in applications. There becomes a need for an “enterprise wallet” to manage all those private keys.

Many mainstream Global 2000 and Fortune 500 enterprises are behind on this, stuck with legacy HSMs focused on encryption keys, and struggling to manage secrets in multi-cloud and services based architectures.

But why don’t HSMs and traditional encryption key management products work for modern secrets management?

There are legacy key management products for encryption, databases, even cloud services in some cases. The difficulty many teams encounter is when these legacy tools are used for devops and cloud infrastructure related keys like API keys, pem files, micro-services secrets, tokens, and others. We are seeing a gap in capability where they are not able to address these needs for a variety of reasons. Here are a few challenges of legacy key management solutions we’ve collected speaking with teams working on a modern cloud-based devops workflow:

Focused on encryption and PKI related keys and certificates, not really API keys
Don’t have cloud-native architectures
Not built for DevOps processes
Lack capability for dynamic secrets generation and rotation with multi-cloud services
Lack capability for containers & microservices

One example of a legacy company trying to plug a hole in its offerings here is CyberArk. They actually purchased an open source secrets management tool called Conjur to add that capability. Here’s how CyberArk announced the Conjur acquisition:

CyberArk aims to revolutionise DevOps security by using the fruits of the Conjur acquisition to deliver an “enterprise-class, automated privileged account security and secrets management” platform to secure the DevOps lifecycle and cloud-native environments. Which is nice. Privileged account credentials — SSH keys, API keys and more — are proliferating throughout IT infrastructure because of DevOps practices. CyberArk has acquired Conjur in an attempt to train and saddle this wild horse of software development.

Other traditional password and key management and encryption vendors haven’t addressed this capability gap — so they’re somewhat behind the 8-ball.

This gap in capability around secrets management for cloud infrastructure & devops is why a lot of companies have built their own secrets management tools in-house and helps explain why there are so many open source approaches — per below. This phenomena of many varied open source solutions to the secrets management problem shows it is a pretty large pain point for modern development organizations.

III. Options for secrets management

No real industry standard exists yet, as everything is fairly early in this space. At CryptoMove, we’re working on an enterprise-scalable solution for secrets management leveraging moving target defense technology under the hood. More on that later.

Due to the absence of too many off the shelf solutions, many companies have tried to build their own secrets management tools. Here are a few:

Square — Keywhiz

Keywhiz is a tool built by Square and operated in production for some time. It has a lot of features for storing and managing application and infrastructure secrets. Square describes Keywhiz as follows:

“Keywhiz makes managing secrets easier and more secure. Keywhiz servers in a cluster centrally store secrets encrypted in a database. Clients use mutually authenticated TLS (mTLS) to retrieve secrets they have access to. Authenticated users administer Keywhiz via CLI. To enable workflows, Keywhiz has automation APIs over mTLS.”

Pinterest — Knox

Pinterest also built its own secrets management tool and open sourced it, calling it Knox (as in Fort Knox). Here were the problems Pinterest was facing which led to the creation of Knox:

“Pinterest has a plethora of keys or secrets doing things like signing cookies, encrypting data, protecting our network via TLS, accessing our AWS machines, communicating with our third parties, and many more. If these keys become compromised, rotating (or changing our keys) used to be a difficult process generally involving a deploy and likely a code change. Keys/secrets within Pinterest were stored in git repositories. This means they were copied all over our company’s infrastructure and present on many of our employees laptops. There was no way to audit who accessed or who has access to the keys. Knox was built to solve these problems.”

Per Pinterest, the goals of Knox are:

“Ease of use for developers to access/use confidential secrets, keys, and credentials
Confidentiality for secrets, keys, and credentials
Provide mechanisms for key rotation in case of compromise
Create audit log to keep track of what systems and users access confidential data”

Lyft — Confidant

Lyft’s approach to secrets management resulted in a tool called Confidant. Lyft keeps it short and sweet on the description: “Your secret keeper. Stores secrets in DynamoDB, encrypted at rest.”

In describing the problem Lyft was trying to solve with Confidant, Lyft explains that with all the new internal and external services, keeping track of keys and secrets became a very laborious process:

“As Lyft has grown, we’ve added numerous services to our infrastructure. These services have credentials to internal services and external services, SSL keys, and other types of secrets. Each of these services has multiple environments, and to insulate these environments from each other, they have a version of each of these secrets for each environment. In many cases some of these secrets may be shared across a few services. Given a large number of services, this leads to a very large number of credentials.
The rotation of these secrets can be a laborious process, especially credentials for external services, since a large number of external services don’t have rotation methods that can be done without some amount of downtime and coordination. Coordination of the rotation of secrets became a difficult and time-consuming process for us pretty early on, and we knew the problem would only get worse as we added more internal services and more external dependencies.”

Here is a discussion of Confidant on Hacker News, with a lot of interesting feedback and compliments.

Many other tech companies have tried to are or are trying to create their own secrets management tool. Name a major tech company and they are probably working on this. More mainstream Fortune 500 or Global 2000 companies often are not aware of the problem yet, but as they increasingly transform their infrastructure and software delivery model, it’s likely they will run into the same issues that led to the creation of Keywhiz, Confidant, Knox, and others.

Docker secrets

Docker secrets management is one of the newer secrets management tools out there. Coincidentally, it was designed by some of the overlapping folks who worked on Keywhiz at Square. Here’s how Docker describes the problem and why traditional static secrets management tools or encryption key management products can’t deal with these issues:

“A critical element of building safer apps is having a secure way of communicating with other apps and systems, something that often requires credentials, tokens, passwords and other types of confidential information — usually referred to as application secrets. We are excited to introduce Docker Secrets, a container native solution that strengthens the Trusted Delivery component of container security by integrating secret distribution directly into the container platform.
With containers, applications are now dynamic and portable across multiple environments. This made existing secrets distribution solutions inadequate because they were largely designed for static environments. Unfortunately, this led to an increase in mismanagement of application secrets, making it common to find insecure, home-grown solutions, such as embedding secrets into version control systems like GitHub, or other equally bad — bolted on point solutions as an afterthought.”

Here’s an architecture diagram of how Docker secrets works:

Here’s an example of how Docker secrets can be used to manage environmental variables as well as AWS credentials. More documentation on Docker secrets is available here.

Vault by Hashicorp

Hashicorp is an open source software company with many products, probably best known for its products like Vagrant and Terraform, and also has a secrets management product called Hashicorp Vault. The New Stack publishes a good overview of the pros and cons of Hashicorp Vault here, which discusses some of the benefits of managing secrets in an external tool vs. environmental variables, working nicely in the Hashicorp ecosystem, and the benefits of API integrations.

There is also an interesting comparison between Hashicorp Vault and Keywhiz from The New Stack:

While all the above is great — as is typically the case with software — certain tradeoffs were made in Vault’s design, and there are some limitations. While there are many benefits to being able to run Vault as a service, this does lead to increased infrastructure costs, and the associated pains of managing that infrastructure. Additionally, not all of Vault’s benefits are available for all use cases. For example, dynamically generated secrets can only integrate with a limited number of other services. Also, Keywhiz by Square is another big player in this area that is worthy of attention. The biggest fundamental difference between Vault and Keywhiz, is that while Vault exposes secrets over an API, Keywhiz uses a FUSE filesystem. Vault wrote a nice and fairly objective writeup on the comparison. Lastly, if you are working with Chef or a related tool, it’s probably easier initially to use their integrated solution than it is to wire it up to Vault.

Due to the complexity of standing up and maintaining a Hashicorp Vault infrastructure, there are quite a few guides and tutorials. Many companies end up hiring consultants or engaging in professional services contracts as part of their Hashicorp Vault rollouts. Here’s a handy guide and tutorial for best practices:

Have you ever installed Hashicorp Vault and wondered to yourself: “Am I actually protecting my organization?” You’re not alone. While it’s easy to install Vault, making sure that it is configured correctly for productivity and security can be a challenging task. I’ve built my fair share of guides and webinars and worked with Vault a-lot recently. This has led me to create my own list of Vault’s best practices.

Other tools out there include one created by T-Mobile called “T-Vault”, which uses some integrations to Hashicorp Vault under the hood and adds a variety of enterprise functionality. One CISO recently commented about the comparison of Hashicorp Vault to CryptoMove’s Tholos Vault in a post:

If you are using, or thinking of using, Hashicorp Vault you should look at Cryptomove. For being an early stage startup the product seemed quiet mature. The concept of constantly moving keys and data was very interesting and which I cannot do justice in a short LinkedIn post.

Torus

Torus is another tool for secrets management. It takes a hosted approach, while open sourcing its clients. There is an interesting HN discussion of Torus here.

Credstash

Credstash is a pretty interesting distributed credential management system that is built on top of AWS KMS. Here’s how it describes the problem it tries to solve as well as its lighter-weight approach to the secrets management toolbox:

Software systems often need access to some shared credential. For example, your web application needs access to a database password, or an API key for some third party service.
Some organizations build complete credential-management systems, but for most of us, managing these credentials is usually an afterthought. In the best case, people use systems like ansible-vault, which does a pretty good job, but leads to other management issues (like where/how to store the master key). A lot of credential management schemes amount to just SCP’ing a secrets file out to the fleet, or in the worst case, burning secrets into the SCM (do a github search on password).
CredStash is a very simple, easy to use credential management and distribution system that uses AWS Key Management Service (KMS) for key wrapping and master-key storage, and DynamoDB for credential storage and sharing.

Sneaker

Sneaker is a tool that uses AWS KMS and S3 to create a lightweight secrets management capability:

sneaker is a utility for storing sensitive information on AWS using S3 and the Key Management Service (KMS) to provide durability, confidentiality, and integrity. Secrets are stored on S3, encrypted with AES-256-GCM and single-use, KMS-generated data keys.

There’s a pretty comprehensive and helpful comparison of various open source secrets management solutions and tools here. In addition to its comparison of tools, the author — who provides consulting on microservices and cloud architecture — lays out a great way to think about secrets best practices:

No secret should be written to disk in cleartext — ever
No secret should be transmitted over a network in cleartext — ever
All secret lifecycle and access events should be recorded in an incorruptible audit log
Secret distribution should be coordinated by a authoritative delegator such as a container/service scheduler or working in a close trust relationship with the scheduler
Operator access to secret cleartext should be limited — if not impossible without subversive efforts
Secret versioning or rolling should be easier to accomplish than revealing cleartext
All infrastructure components related to secret management and distribution should be mutually authenticated
Secure system configuration should be easier than advanced, and likely insecure configuration
The attachment of a secret to a service or container should be protected by rich (pluggable) access control mechanisms — role based access control is a plus
Anything that can be done to minimize the value of a secret should be done

Any time so many open source projects pop up in a space, it is a pretty good indication that there is a common problem lots of developers, security professionals, and software organizations are running into. Open source solutions and tools are often the first to sprout up.

Cloud vendor approaches

Cloud infrastructure vendors have noticed that their users and customers are struggling with secrets management, and have started developing their own generic solutions, like AWS Secrets Manager, Azure Key Vault, and others described below. In some ways, these products are reminiscent of Amazon’s Whole Foods 365 “Lacroix-killer” or Safeway’s famous “Select” brand of sodas and other private label products. In the cloud world we live in, it seems reasonable to expect cloud-generic versions of such products and others to pop up.

AWS

AWS Secrets Manager is a tool that enables AWS users to manage secrets and credentials rather than saving them on disk or using one of the KMS-backed credential management open source solutions, like Sneaker. Here’s a video from the AWS product manager on how Secrets Manager is supposed to work:

There’s a handy discussion of comparisons between AWS Secrets Manager and one of the open source solutions, Hashicorp Vault, on reddit. Some of the limitations of AWS Secrets Manager (according to a person from Hashicorp on Reddit) include: lack of zero knowledge for admins, dynamic secret generation, one-click/API call certificate generation, SSH CA authority, encryption as a service, cross-region/cloud/datacenter replication, control groups for multi-person approvals, pluggable architecture, and some others.

AWS Parameter Store. AWS parameter store can be another way to manage secrets, without leveraging Secrets Manager. Here is a guide for how to set up AWS Parameter Store for secrets management.

AWS KMS. AWS KMS is not really a secrets management tool, although it is possible to wrap secrets with encryption using KMS, which is what some of the open source solutions referenced above do with KMS.

Azure

Azure key vault is a service that enables customers to manage all secrets (keys, certificates, connection strings, passwords etc) for their cloud application in a single place. It is integrated out of the box with sources and destinations of secrets in Azure, but can also be used by applications outside Azure.

It is sort of a combination of AWS KMS and AWS Secrets Manager, both supporting encryption services within Azure as well as managing non-encryption secrets and credentials. Here’s how Microsoft describes the problems Azure Key Vault is meant to solve:

Azure Key Vault helps solve the following problems
Secrets Management — Azure Key Vault can be used to Securely store and tightly control access to tokens, passwords, certificates, API keys, and other secrets
Key Management — Azure Key Vault can also be used as a Key Management solution. Azure Key Vault makes it easy to create and control the encryption keys used to encrypt your data.
Certificate Management — Azure Key Vault is also a service that lets you easily provision, manage, and deploy public and private Secure Sockets Layer/Transport Layer Security (SSL/TLS) certificates for use with Azure and your internal connected resources.
Store secrets backed by Hardware Security Modules — The secrets and keys can be protected either by software or FIPS 140–2 Level 2 validates HSMs

Here is a video of Microsoft’s product manager on Azure Key Vault describing how it’s supposed to work:

Google

Google has an encryption key management service, but lacks a secrets manager product. The closest to come in Google cloud is to use KMS to wrap secrets with encryption and then store them using other GCP services that store secrets elsewhere. Here is Google’s documentation on how to do this. Also, here’s an interesting tutorial on how to use a combination of Google Cloud Platform services to store secrets in serverless application architecture. Google also has a general guide on ways to store secrets, ranging from in storage to in code itself (is this a great idea?).

Newish startups focused on this space

Apart from the open source tools and cloud-vendor products, there are a few startups focused on building products in the space of key and secrets management.

Envkey is a recent startup launched out of YCombinator. It has good UI & UX, better than most of the secrets management products that are more focused on CLI and APIs.
UnboundTech (formerly known as Dyadic) is based in Israel. They use multi-party computation for key management, and are backed by strategic investors including Citibank and Goldman Sachs.
Fortanix calls its tech “SDKMS” or software-defined key management systems. It is integrated heavily with the Equinix data center company, and is based on Intel SGX technology (and also backed by Intel as a strategic investor).

How does CryptoMove fit into all this?

At CryptoMove, we started out with a technology for moving target data protection for any type of data. As we’ve productized the underlying technology, we’ve seen prototypes of a variety of use cases ranging from protection of keys, secrets, files, databases, even live-streaming video or data across drone swarms, embedded devices such as cameras or sensors, and robotics.

However, after working with many early design and R&D partners across the Fortune 100, as well as federal agencies like the Department of Homeland Security, NIST, we realized that by far the most common first use case for CryptoMove was trying to apply our technology to key and secrets management. Armed with this learning, we spent over a year talking with hundreds of security teams, devops teams, developers, CISOs, and others to learn about the pain points around existing key and secrets management tools and processes especially as applied to multi-cloud, microservices, and IoT infrastructures.

Based on what we’ve learned, we have been working on a cloud-native SaaS secrets management product we call CryptoMove Tholos (Tholos being the greek word for Vault), with moving target data protection technology under the hood. More information is available here on our website and we’ll continue to blog about it.

IV. Important objectives for key and secrets managers and vaults

Whether a commercial solution, open source project, or homegrown tool, there are a few common objectives for key and secrets managers and key vaults.

Management of the secrets. Of course, the basic tasks of secrets generation, rotation, revocation, assignment, and sharing are important. Those are often laborious processes in modern software organizations with many internal and external services. It’s important that a secrets management tool can make the processes associated with working with secrets easy and organized.
Authentication & machine identity. It’s important that a secrets management tool have comprehensive and accessible APIs. The most common use case for secrets management in modern devops and cloud-native workflows is to authenticate application components and services. Especially as services and business processes are increasingly automated, machine-to-machine authentication and identity via secrets management is critical.
Storage of the secret. Here is an area where there actually hasn’t been much innovation. Most secrets tools today focus on inventory and policies for generation and rotation. In terms of key and secret storage, most common practices are dropping keys into a database, storage, or some other back-end, wrapped in encryption. Here’s where things like moving target defense, fragmentation, and decentralized or distributed storage of keys secrets can be an interesting way to approach things.
Reliability of the secrets management infrastructure. Secrets should not just be well organized, inventoried, accessible, and stored, but they should also be managed in a system that is highly available and resilient.

V. Challenges for secrets management in devops / cloud transformations

Secrets management is a problem many organizations encounter, as the examples from Square, Lyft, Pinterest, T-Mobile, and others show. Outside of the general issues around managing secrets for cloud and services based systems, what are the specific challenges organizations face around secrets?

Prioritization. In some organizations simply prioritizing secrets management is a challenge. It’s not always clear that the problem is happening, especially if the security team is more involved with corporate security and securing endpoints rather than in the secure software development lifecycle. Many organizations still operate in the framework where HSMs are the only option, and often encryption keys are the only secrets really considered. Typically this is due to cloud or services based transformations still in their early stage.
Multi-infrastructure approaches. Secrets management across infrastructures can be a challenge. Would an organization use AWS Secrets Manager to manage secrets related to Azure storage? Or Azure key vault for AWS-related secrets? Some of the open source key and secrets management solutions may be a multi-cloud approach here, as well as enterprise SaaS vaults.
Scalability and orchestration. Autoscaling secrets management infrastructure is important, especially as services operate at massive scale. Secrets management tools should also orchestrate well with other parts of the devops stack and with other tools in the security stack as well.
IAM integration. Secrets management tools should integrate with various authentication options, like Okta, OneLogin, Active Directory, AWS IAM roles, etc. At CryptoMove, we’ve done integrations with a few of the prior listed, as well as Auth0, which we love.
UX/UI. Good APIs are important for secrets management, but as seen by many of the teams we’ve spoken with, good UI and UX is also very important. Not all keys and secrets can be programmatically generated, especially with certain legacy systems. It’s important to provide a solid interface for admins and policy creation. Because some of the open source tools did not focus too heavily on UIs, certain developers have tried to make their own on top. They’re not always fully supported but can be found with a bit of creative GitHub searching.
Root of trust. There’s always the classic “turtles” problem of how to initiate the secrets management system itself. There’s a great talk from Netflix on how they deal with the turtles issue from last year’s 2017 Enigma conference. Ultimately, there will be some root of trust, whether an HSM, something like Shamir Secret Sharing for unsealing, multi-party computation, or possibly other approaches.

Discussion of how Netflix bootstraps its secret management at scale

Who owns it? DevOps or Security? The responsibility and ownership of secrets management in an organization is often not an easy question to answer. Often the devops team and hands on engineers are dealing with testing, integration, & deployment related tools and processes. Security teams are typically involved in architecture, monitoring things, tracking, and security awareness. Would developers manage their own secrets, or should security teams manage the secrets? A bit of both? We’ve seen some organizations where security is heavily involved in secrets management and some where they are not.
Discovery. Just knowing where all secrets are is a really hard problem. Many organizations struggle with asset inventory around secrets, which makes knowing and risk-modeling the secrets attack surface difficult.
Audit & usage tracking. Knowing when secrets are used and accessed can be helpful during incident response scenarios, as well as to help discover insider threats. The engineers who built Docker secrets manager, previously at Square, discuss this idea of “crypto-anchoring” to make exfiltration harder here. Here’s a video explanation of crypto-anchoring:

Developers who worked on secrets management at Square & Docker explain their approach

Infrastructure setup and management. Any organization that uses even a few cloud services or internal services in their development has secrets to manage. But not all organizations have the same level of resources when it comes to standing up and maintaining a secrets management infrastructure. How should it be set up? As a SaaS service? Individually in different environments and infrastructure? Should it take a team or services? These are all important questions that can depend on the project and type of organization.

Questions to ask when thinking about secrets management

Like with any technology transformation wave, there is a maturity model of organizations with various stages in the secrets management journey. Wherever an organization is on the maturity model and cloud-native/devops transformation, here are some helpful questions to think about when designing, implementing, upgrading, or optimizing a secrets management workflow and infrastructure.

Where are the secrets now? How are they managed today?
What secrets do we want to manage? Cloud-services? Cloud-native applications? Containers and microservices? Data analytics and data lakes? Conventional databases? IoT? Private keys to crypto or blockchain applications?
Who will the users be? Security team? Devs? Admins?
How will the secrets management infrastructure be managed? With a dedicated team? Hosted as a service?
Is a UI or API needed? Combination of both?
What is the multi-cloud or multi-infrastructure secrets management strategy? Should cloud service provider tools manage their own secrets? Should an organization spread secrets across multiple tools? Infrastructures?
What other tools are managing secrets now? Are there any? Is there currently a secrets management strategy in place or awareness among the team?
What is the back-up and resiliency strategy for secrets?
Should the organization leverage open source tools or commercial enterprise focused tools, or a combination of both?
How are the secrets secured at rest? Beyond management, how are secrets actually protected? On storage? In a database? Some other way?
Who should have access to the secrets? This can be a complex question with varying answers for varying workflows or secrets.
What policies should apply to secrets management? How will versions be tracked? Revocation? Time-based policies?

Secrets management is a complex and emerging problem presented with the transformation towards cloud-native, services, and IoT infrastructure. As can be seen with the most advanced technology companies like Square, Pinterest, Lyft, and many others, every modern software organization at some points hits the wall when it comes to secrets management.

The exciting thing about secrets management is it sits at the intersection of data security and application security, opening up new ways to think about risk management and threat modeling an attack surface. And when done correctly, secrets management at scale can accelerate business process change around cloud and services transformations.

One thing is clear — we are in the early days of the curve when it comes to secrets management with a lot of room for innovation ahead.