With Great Secrets Come Great Responsibilities
An introduction to HashiCorp Vault
By Yashil Kalyani and Ananya Rajagopal
The enterprise IT scene is rapidly evolving, with a push to migrate to the cloud. Sensitive data is generated in epic proportions, with remote and mobile work being the new norm in the post-pandemic world. As organizations continue to build complex, cloud-based operating models, they face an interesting yet long-standing challenge:
How do they keep secret data … secret?
Managing enterprise secrets is challenging and demanding
What are these “secrets” I’m referring to?
In this context, we’re talking about managing a set of credentials that grant a user — or a machine — in an enterprise environment the appropriate authorization and access to a system (e.g., usernames and passwords, database credentials, API tokens, and TLS certificates).
Why is managing secrets challenging?
As a child, I thought setting my password as “password” was very clever; until my mom cracked it in about five seconds and discovered that I used her credit card to purchase several comics! Organizations face much larger impacts when secrets are not managed well. For example, we’ve all seen employees maintain spreadsheets or several sticky notes to keep track of their passwords (i.e., secret sprawl). This can lead to credentials being leaked in the source code or config management script, making it hard to audit who used which secrets. When it is time to rotate credentials, the problem is exacerbated.
It isn’t just humans who leak secrets — applications can to. They can share sensitive information via:
- stdout, which gets logged in the log management system
- diagnostic output (e.g., traceback)
- application monitoring tools
It’s often difficult for applications to implement sophisticated cryptography correctly. It is easy to get wrong and can compromise the whole system.
Managing secrets with HashiCorp Vault
HashiCorp describes Vault as:
An identity-based secrets and encryption management system. It provides encryption services that are gated by authentication and authorization methods to ensure secure, auditable, and restricted access to secrets. It is used to secure, store, and protect secrets and other sensitive data using a UI, CLI, or HTTP API.
Vault’s key features tackle the challenges of secrets management head-on.
Centralization
Vault tackles the sprawl problem by providing a centralized store for all your secrets. It encrypts secrets before writing them to persistent storage. Encryption takes place at rest and in transit. It further enables access control and provides an audit trail.
Dynamic secrets
Vault provides ephemeral credentials that enable applications to use secrets effectively, creating a moving target for attackers, even if the application exposes a secret. Each application machine is given a unique secret, helping to pinpoint compromised machines.
Secrets revocation
After creating dynamic secrets, Vault will also automatically revoke them when the lease is up — and can revoke multiple secrets at once (e.g., all secrets belonging to an application or user group). Because dynamic secrets are unique for each user or machine, the impact of attacks can be minimized by quickly revoking the secret.
Encrypt as a service
As the name suggests, Vault makes encryption easy by abstracting away the complexity involved. It enables the creation of “named” keys whose identifiers can be shared without sharing the key itself. Vault only exposes high-level APIs (i.e., encrypt/decrypt, sign, and verify) to applications. The implementation is shielded from the application. Key lifecycle management is also provided.
The core pillars of Vault (and how they address enterprise security concerns)
Authentication
This ensures secure access to Vault. Users supply identity information using auth methods, helping Vault validate who they are. It can validate identity using third-party trusted sources, including AWS, Active Directory, or native auth methods (e.g., username and password).
Access
Vault grants access by issuing a token based on policies associated with the authenticated user identity. Each token has a policy attached to it that determines which actions can be performed, enabling Vault to control who has access to what.
Secrets engine
The “bread and butter” of Vault — this is where the magic happens. The secrets engine handles and stores different types of secrets. HashiCorp defines secrets engines as:
… components which store, generate, or encrypt data. Secrets engines are incredibly flexible, so it is easiest to think about them in terms of their function. Secrets engines provide some set of data, they take some action on that data, and they return a result.
Some secrets engines simply store and read data — like encrypted Redis/Memcached. Other secrets engines connect to other services and generate dynamic credentials on demand. Other secrets engines provide encryption as a service, TOTP generation, certificates, and much more.
Let’s bring the pillars together in an example. Consider an app running on EC2 that wants to access a PostgreSQL database with dynamic credentials. This is what the workflow will look like:
- The EC2 machine authenticates itself and acquires a token with the appropriate policies attached (for our example, assume that the policy grants the required read/write permissions).
- EC2 calls Vault APIs using the token, and Vault validates the token.
- Vault interacts with the database to create dynamic credentials and passes them back to the application running on EC2.
- The EC2 instance uses the credentials to communicate with the database.
- When the credentials expire, the process is repeated.
In this example, the application used the AWS auth method to authenticate against Vault. It then used the token to interact with the Postgres secrets engine to supply on-demand credentials.
Planning your Vault implementation
I recommend a three-layer approach to your Vault implementation. Think about implementing Vault like building a house. You buy land and build the exterior, design the interior, and then live in it. Similarly, you create infrastructure and deploy Vault, configure Vault, and then use it.
Layer 1: Create infrastructure and deploy vault
HashiCorp provides a hosted Vault solution, HCP Vault. Consider this option if deploying custom infrastructure is not preferred. When creating self-managed infrastructure, consider the following architectural pattern to get started quickly with a single cluster of Vault:
When you are ready for an enterprise-wide implementation, I recommend leveraging HashiCorp’s multi-cluster architecture guide.
The major public cloud platforms also provide quick-start guides (AWS, Microsoft Azure, and Google Cloud) to support you on your journey.
Layer 2: Vault configuration
One of the pillars behind the Tao of HashiCorp is automation through codification. It is strongly recommended to codify the configuration of Vault using Terraform, a popular infrastructure-as-code tool.
Vault configuration is a vast topic that requires taking a holistic view of your people, organizational processes, and technical implementation. Key topics to consider when configuring Vault are listed below.
Authentication methods, such as:
- User lifecycle management
- Different authentication needs (remember that a human user’s authentication journey will differ from that of a machine)
- Application specifications
Static versus dynamic secrets (and your secrets schema), including:
- Secrets engines to be supported
- Start with the key/value (KV v2) secrets engine, and implement others as the Vault management matures
Policies and considerations, such as:
- Follow the principle of least privilege
- Codify policy creation and utilize ACL (access-control lists) policy path templates and a version control system
Lastly, secret consumption and considerations when planning this strategy.
Layer 3: Using Vault
Expose Vault as an internal product to the organization. Roll out “features” as the product matures. The most successful implementations of Vault start with the user experience and leverage design thinking (e.g., empathize with the user, create a detailed user journey and product plan) to enable a smooth transition to Vault.
Popular Vault usage patterns
Using Vault with Jenkins
Jenkins reads secrets using the Vault plug-in. As specified in the documentation, this plug-in allows authenticating against Vault using the AppRole authentication back end. HashiCorp recommends using AppRole for servers / automated workflows (like Jenkins) and tokens (default mechanism, GitHub token, etc.) for every developer’s machine.
Using Vault with AWS Lambda functions
HashiCorp strongly recommends using the Vault Lambda extension to retrieve secrets from Vault. With the Vault Lambda extension in your Lambda at runtime, and setting the proper environment variables, the function can retrieve secrets from Vault before it executes. Check out this learning guide for a detailed tutorial.
Using Vault with Kubernetes
Vault supports several Kubernetes integrations including:
- Vault Agent provides access to Vault secrets by deploying a Vault-agent sidecar into a Kubernetes Pod.
- Vault CSI pattern fetches secrets stored in Vault and uses the Secrets Store Container Storage Interface (CSI) driver interface to mount them into Kubernetes Pods.
- Vault Secrets Operator for Kubernetes provides a native method to retrieve and sync Kubernetes secrets, including post-rotation updates.
Review the documentation to determine which Kubernetes integration pattern best suits your needs.
A hard lesson learned
Perhaps two of the most important considerations in your Vault implementation include the principle of least privilege and properly saving the root and unseal keys upon initialization.
Once, in the early stages of implementing Vault, there was a need to reconfigure some aspects of the secret structure. Initially, the root key (which had unlimited access to the Vault instance) had been shared across multiple developers. As a result, the root key had been used to re-initialize Vault (unintentionally) and the unseal keys were not saved. The Vault instance locked itself and was unusable by the developers and other applications. This underscores the vulnerability and risk in sharing root keys without a clear understanding of the potential consequences. In this case, access to the secrets data of the organization was irreversibly revoked.
This incident emphasized the need for more stringent procedures in the hands of the primary administrator and across the entire development team. After this, the organization adopted the principle of least privilege and developers no longer had access to the root key. Instead, LDAP and AppRole auth methods were introduced, with specific policies catering to each development user group.
Other leading practices
There are several key capabilities to use while implementing Vault in your system:
Tokens and dynamic secrets
Organizations can generate short-term authentication tokens, reducing the risks associated with long-lived credentials. Dynamic secrets can automate secret rotation, further increasing the secret posture of the organization.
Access control policies
Vault enables organizations to define specific access policies, restricting user and application access to certain secrets. These policies ensure that only authorized entities can retrieve or modify secrets in alignment with their designated roles and responsibilities.
Secret encryption at rest
This is applied automatically by Vault, so when you store secrets, they’re encrypted before being persisted to storage. Even if unauthorized access does occur, no applications or users will be able to read the secrets.
Integrate with LDAP or Active Directory
By connecting with identity providers such as LDAP or Active Directory, organizations can centralize and manage authentication. Implementing identity-based policies also enables the organization to follow the principle of least privilege across the entire enterprise.
Finally, as you are planning your Vault implementation, keep a few things in mind:
- Avoid storing sensitive data outside Vault, such as in configuration files or version control systems — it undermines the centralized and secure nature of Vault.
- Limit the use of root tokens, and prioritize the use of tokens with appropriate policies for day-to-day operations. Improperly using the root token for routine operations is a huge security risk.
- Plan for and implement routine audit reviews; this is crucial to ensuring compliance and promptly detecting any unusual activities.
- Establish a disaster recovery method for the Vault instance to prevent data loss and unnecessary downtime.
Conclusion
In an ever-changing IT landscape, where the safeguarding of secrets is paramount, HashiCorp’s Vault emerges as an effective secrets management tool. Whether you are struggling to address the challenges with secrets management, considering implementing Vault, or just looking for practical tips you can use in your Vault implementation, we invite you to continue exploring Vault’s role in your secrets management journey.
Slalom is a next-generation professional services company creating value at the intersection of business, technology, and humanity. Learn more and reach out today.
