Managing Secrets Using Hashicorp Vault

by Stenio Ferreira

Overview

Managing secrets throughout the lifecycle of an application is not a trivial task — if you overly restrict access, implementations can become too complex. On the other hand if you have a lax approach, you will increase the risk of a security breach.

Additionally, other concerns should also be addressed while managing the secrets:

• How to handle external and internal threats (such as hackers and disgruntled employees).
• How to securely share secret management responsibility among a set of stakeholders.
• How to allow seamless access to both humans and services.
• How to audit who accessed a secret at a certain date and time.

Hashicorp Vault is an open source secret management and distribution tool that proposes an answer to these and other questions. In this blog post I will introduce the technology and provide a walkthrough, through a Proof of Concept example, describing how to install, initialize, and access the secrets using token authentication. This walkthrough provides an example closer to enterprise reality than the one found in the official tutorial, while not going into details of more complex implementations. At the end of the post I list suggestions of areas in this implementation that can be further enhanced, with associated reference links.

Hashicorp Vault

Hashicorp Vault is developed and maintained by our friends from Hashicorp, famous for making delightful and popular tools such as Vagrant, Consul and Terraform. Hashicorp Vault follows the same guiding principles as the company’s other solutions, resulting in a product that is simple to use due to its modularity, yet powerful and flexible on how it can be implemented. There are many ways of describing Hashicorp Vault’s features, but Seth Vargo, one of its maintainers, summarizes it in this interview by citing four main “pillars”:

• Mitigation of internal and external threats: allowing Vault to be sealed/unsealed in response to crisis, and supporting a variety of security backends.
• Dynamic secrets: Vault is able to generate credentials automatically for different backends.
• Lease renewal/revoke access to secrets: Vault allows you to specify TTL rules for a secret, allowing fine control over permissions.
• Auditing: Once enabled, every secret request will be logged, with the output directed to a file or to syslog.

Having said that, is Vault truly secure? Besides being available as open source, the tool has undergone 3rd party security audit by iSEC, with details discussed here and here.

Backends

Vault offers modular plug-in for three main areas — encrypted secret storage, authentication controls and audit logs:

• Secret storage: This is the solution that will “host” the secrets. Available backends include AWS S3, Consul, Generic (file storage) among others
• Authentication controls: The authentication mechanism used. Available backends are AWS EC2, LDAP, Github, Tokens, Username/password, and others
• Audit logs: Where logs are sent. Available File or syslog

The specific mix of backends will depend on your project’s requirements and constraints, and more than one backend can be enabled. Additional information on each backend, including API request commands to access them, can be found in the official documentation.

Proof of Concept:
Accessing Secrets Using Token Authentication and Response Wrapping

One possible scenario while using Vault is when you have a set of clients, that are previously defined and need access to secrets. For example, in order for Mary the developer to access a secret using her Github account, a Vault manager would need to enable the GitHub backend, and based on the associated policy, Mary would only need to authenticate to Vault using her GitHub credentials. A similar scenario would take place for sample-app-service, which has an AWS IAM-based service account and requires access to certain secrets.

However, recently I worked on a client project that had a different requirement — using a combination of Jenkins, Chef and Microsoft’s Hyper-V manager, a json file was used to describe VMs that should be dynamically created in the Hyper-V host. Given the dynamic nature of these VMs, and the client’s preference for non-cloud solutions, it would be less than ideal to have to hardcode VM names in advance in order to create Vault access credentials.

In this section I will describe a PoC using Vault’s token authentication backend to address the above scenario. Three participants will work in this PoC:

• A VM where Vault is installed
• A VM representing the client that wants access to a secret
• My workstation, which will be acting as a Trusted Entity. The Trusted Entity will be responsible for interacting with most of the Vault API, and passing the secrets to the end consumer. In the real world this role can be played by Jenkins, Terraform, or another CI/Orchestration tool.

One critical problem that this PoC will address is how to pass the access token from the Trusted Entity to the client VM, while lowering the risk that it could be intercepted along the way. This will be accomplished by using Cubbyhole’s response wrapping — a feature that allows a Vault response to be associated with a single use, time-limited token, stored within Vault. The Trusted Entity will only receive this token, and when the client VM uses it to retrieve the access token, it will expire — thus ensuring that the access token was only exposed once.

Sequence diagram of what we plan to accomplish

The code for this PoC can be found here.

Installation

In order to run this PoC, you will need to have:

• Vagrant, a Virtual Machine manager. Instructions for different platforms can be found here.
• Git, a version-control system. Instructions here.

Starting VMs

Once both are installed, open the cmd prompt and execute:

This will start the Hashicorp Vault VM and the Client VM that needs access to the secrets.

Initializing Vault

Before you can execute instructions against Vault such as creating secrets and access tokens, you need to initialize it. Run the following in the command prompt in order to connect to Vault using Vagrant:

Once inside Vault, execute the following to initialize Vault with a given configuration file:

This will output 5 unseal tokens, which need to be used in order to “unseal” or “seal” the Vault. Secrets can only be accessed when Vault is “unsealed”, and the configuration used for initialization defaults to minimum of 3 tokens to allow unseal.

In a real world scenario, these tokens would be distributed among key stakeholders, guaranteeing distributed responsibility for secrets management. They should be stored safely in the same way as personal passwords.

There will also be an additional token in the output — this is the root access token, with full permission to Vault. It should only be used for initial configuration, while recurring operations should be done by policy-constrained tokens.

With the tokens available, execute this command three times, entering one of the provided unseal tokens:

Before performing any operations on an unsealed Vault, you need to login first. This can be accomplished by executing:

As a best practice, we will enable auditing log:

You can also check the default secrets folders by executing:

This will return /cubbyhole, /secrets and /sys, with descriptions of their intended use. For this PoC we are going to use all three of them.

Adding Policies

The git repository includes three access policies, with the idea of mapping an enterprise’s environment secrets to different folders. First we create the policies by executing:

Writing Secrets

Now that we have the policies in place, let’s add some sample secrets to each of the mapped environments:

Create an Access Token Associated with a Policy

Here is where we will create the token that the Node VM will use to communicate with Hashicorp Vault. Since this will be created by the Trusted Entity (i.e. Jenkins), we will use the wrapper pattern to prevent anyone other than the Node VM itself and Vault from knowing the access token. By passing the -wrap-ttl parameter during token creation, Vault returns a Cubbyhole wrapped token, with a specified time-to-live and that is single use. The idea is for once it is used, it returns the access id token and lease, and can’t be used again. Therefore:

This creates an access token and lease with “production” policy applied to it, and stores this information on the Cubbyhole.

Optional parameters

If additional control is desired, a few other parameters can be used with token-create:

• explicit_max_ttl: Sets the time after which this token can never be renewed.
• num_uses: Number of times this token can be used. Default is unlimited.
• renewable: If token is renewable or not. Default is true.
• ttl: Time token is valid. Default is 720hs.

Send Single-Use Token to VM

Different approaches can be taken on how to send the single-use token to the Client VM. In our PoC we will do a simple remote copy by copying the “wrapping_token” value from the above output, opening another terminal window and issuing:

Client VM Gets Access Token

Now that the Node VM has the single access token, open a third console window and execute:

If returns permission denied, token is either expired or has been compromised. Trusted Entity should be notified to contact Vault to revoke that token and create a new one.

If successful, it will return the output:

In the background, what this has done is it has assigned the following json to the ‘RESPONSE’ variable:

In order to parse this json response, you can use regex or the jq tool command line json processor.
Using jq:

This will store the access token in the ACCESS_TOKEN variable in memory.

Client VM Communicating with Vault

Now anytime Client wants to communicate with Vault, it should use the value in the $ACCESS_TOKEN environment variable, for example:

This will return the json:

This token has ttl and will expire in 720hs. In order to keep alive, at regular intervals the Client VM must issue:

Lease renewal is only possible if token still valid. If expired or revoked, notify Trusted Entity and go back to step 1.

Conclusion

Congratulations! You now have a functioning Hashicorp Vault implementation using token authentication.

Where to Go From Here

• Distribute unseal tokens securely using Keybase or PGP, as documented here.
• Put vault behind a SSL web server proxy, as described here.
• Explore the different Authentication and Secret Storage backends.
• Support High Availability mode by integrating Vault with Consul, Zookeeper or etcd.
• Enable dynamic secrets, such as the AWS example described in the official documentation.

Getting Help

Hashicorp Vault has reasonable documentation for each API endpoint and CLI command available online. However for specific parameter usage, you might benefit by issuing the following command:

This is an automatically generated doc that lists the available endpoints, with the associated regex that will trigger each.

To verify a specific path, just use it in the command, such as in the example:

About the Author

Stenio Ferreira is a Solution Architect with Slalom’s Cross-Market delivery center in Chicago. He’s passionate about helping companies introduce DevOps best practices to their internal processes, like creating CI and CD pipelines, describing infrastructure as code, and executing cloud migrations. Follow him on Twitter @stenio123.