Building a Local HashiCorp Vault Cluster — Volume 1

Let the infrastructure team figure out the deployment while we build out the policies

Erik R. Rygg
Feb 6, 2018 · 4 min read

HashiCorp Vault is a sweet little product that can do all sorts of super cool, super secret management, but a production-level deployment is a bit of a task. Depending on how you want to store your secrets and how fault tolerant you want your clusters to be, it could take upwards of 8 or more instances (be it container, virtual machine, etc.). And that will just get you a Vault to work with, then there’s a daunting task of building policies, integrating backends, adding audits… my head hurts already.

In order to tackle this problem it makes sense to parallelize a little bit here. Let the infrastructure folks figure out the best way to architect and manage the deployment while we figure out how the heck Vault actually works within our organization. In order to do that easily, we can just spin up a simple local environment and get to work right away on our policy buildout.

Deploy Dev Server Instance and Setup a Client

Firstly, we are going to deploy a couple Docker containers, so we need to setup a Docker network to get these containers talking to each other. Assuming you already have Docker setup on your dev machine (we’re using Docker for Mac) it’s as simple as:

MY-MAC$ docker network create vault-net
<vault-net ID>

Alrighty, now we have the network we can use to connect our containers.

To get the Vault server instance running, we’ll pull down HashiCorp’s Vault image and run it:

MY-MAC$ docker pull vault

Note we are also connecting to the vault-net network. In a separate terminal (ensure you have the vault binary installed on your local machine) run:


Sweet! We have a dev Vault instance running in a container and we can connect to our instance outside the container. Now what?

We’re going to build another container that we can use as a Vault client to connect via SSH using a One Time Password (OTP).

One Time SSH Passwords

Ok, now we are going to setup our vault instance to deal with ssh using an OTP. We use this as a way to manage shared users such as the default ubuntu user on our ephemeral instances. This gives us a way to still ssh onto our nodes without propagating all the users to the nodes.

Server Setup
Firstly, we’ll need to mount the backend on our local machine:

MY-MAC$ vault mount ssh
Successfully mounted 'ssh' at 'ssh'!

Now we need a role to provide OTP to the clients. We’ll allow the ubuntu user on our client node to use this role.

MY-MAC$ vault write ssh/roles/otp_role key_type=otp  default_user=ubuntu cidr_list=

Client Setup
We’ll need a node we can use to ssh to for our test, so let’s go ahead and setup a simple docker ssh container (note: this is a container I built specifically for this demo, it’s just a sshd container with sshd/PAM configured as well as vault-ssh-helper installed and you can check out the internals here). First, prune them since we are explicitly naming this container.

MY-MAC$ docker container prune -f
Total reclaimed space: XB

Cool, now we’ve got a client running attached to our vault-net network. But, the container is configured with PAM to connect to Vault and we don’t have any local users that are able to authenticate with local passwords. So we’ll use the ubuntu user to authenticate with the vault service.

Docker networking defaults to using the subnet, so the vault instance should be and the client should be

To get the client to connect to Vault with ubuntu set the same env vars we did earlier and ssh to the box from our localhost. First, get the OTP:

MY-MAC$ vault write ssh/creds/otp_role ip=
Key Value
--- -----
lease_id ssh/creds/otp_role/<ID>
lease_duration 768h0m0s
lease_renewable false
key <Password>
key_type otp
port 22
username ubuntu

Now try to ssh using the <Password>

MY-MAC$ ssh ubuntu@localhost -p <random port>
Password: <Password>
Welcome to Ubuntu 14.04 LTS (GNU/Linux 4.4.0-101-generic x86_64)

Nice! We’ve got OTP working with ubuntuusing Vault roles/creds and the vault-ssh-helper! Pretty cool, but even cooler would be using Vault as a certificate authority to store and distribute signed ssh keys for individual users. Stay tuned, we’ll explore that in the next iteration of my Vault journey!

Image Credit: Kristian Hoffer

Rigged Ops

Assembling DevOps

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store