Building a Local HashiCorp Vault Cluster — Volume 1
Let the infrastructure team figure out the deployment while we build out the policies
HashiCorp Vault is a sweet little product that can do all sorts of super cool, super secret management, but a production-level deployment is a bit of a task. Depending on how you want to store your secrets and how fault tolerant you want your clusters to be, it could take upwards of 8 or more instances (be it container, virtual machine, etc.). And that will just get you a Vault to work with, then there’s a daunting task of building policies, integrating backends, adding audits… my head hurts already.
In order to tackle this problem it makes sense to parallelize a little bit here. Let the infrastructure folks figure out the best way to architect and manage the deployment while we figure out how the heck Vault actually works within our organization. In order to do that easily, we can just spin up a simple local environment and get to work right away on our policy buildout.
Deploy Dev Server Instance and Setup a Client
Firstly, we are going to deploy a couple Docker containers, so we need to setup a Docker network to get these containers talking to each other. Assuming you already have Docker setup on your dev machine (we’re using Docker for Mac) it’s as simple as:
MY-MAC$ docker network create vault-net
Alrighty, now we have the network we can use to connect our containers.
To get the Vault server instance running, we’ll pull down HashiCorp’s Vault image and run it:
MY-MAC$ docker pull vaultMY-MAC$ docker run --network vault-net --cap-add=IPC_LOCK -e 'VAULT_DEV_ROOT_TOKEN_ID=my_root_token_id' -p 8200:8200 vault<Logs and stuff will show up here>
Note we are also connecting to the
vault-net network. In a separate terminal (ensure you have the
vault binary installed on your local machine) run:
MY-MAC$ export VAULT_ADDR=http://0.0.0.0:8200MY-MAC$ export VAULT_TOKEN=my_root_token_idMY-MAC$ vault status
Seal Type: shamir
Key Shares: 1
Key Threshold: 1
Unseal Progress: 0
Cluster Name: vault-cluster-b631b373
Cluster ID: 2814a88c-4074-9122-3f9d-f5e81d7e8fc1High-Availability Enabled: false
Sweet! We have a
dev Vault instance running in a container and we can connect to our instance outside the container. Now what?
We’re going to build another container that we can use as a Vault client to connect via SSH using a One Time Password (OTP).
One Time SSH Passwords
Ok, now we are going to setup our vault instance to deal with ssh using an OTP. We use this as a way to manage shared users such as the default
ubuntu user on our ephemeral instances. This gives us a way to still ssh onto our nodes without propagating all the users to the nodes.
Firstly, we’ll need to mount the backend on our local machine:
MY-MAC$ vault mount ssh
Successfully mounted 'ssh' at 'ssh'!
Now we need a role to provide OTP to the clients. We’ll allow the
ubuntu user on our client node to use this role.
MY-MAC$ vault write ssh/roles/otp_role key_type=otp default_user=ubuntu cidr_list=172.18.0.0/16
We’ll need a node we can use to ssh to for our test, so let’s go ahead and setup a simple docker ssh container (note: this is a container I built specifically for this demo, it’s just a sshd container with sshd/PAM configured as well as vault-ssh-helper installed and you can check out the internals here). First, prune them since we are explicitly naming this container.
MY-MAC$ docker container prune -f
Total reclaimed space: XBMY-MAC$ docker run -d -P --name vault_ssh_client --network vault-net errygg/vault-ssh-helperMY-MAC$ docker port vault_ssh_client
22/tcp -> 0.0.0.0:<random_port>
Cool, now we’ve got a client running attached to our
vault-net network. But, the container is configured with PAM to connect to Vault and we don’t have any local users that are able to authenticate with local passwords. So we’ll use the
ubuntu user to authenticate with the vault service.
Docker networking defaults to using the 172.18.0.0/16 subnet, so the vault instance should be 172.18.0.2 and the client should be 172.18.0.3.
To get the client to connect to Vault with
ubuntu set the same env vars we did earlier and ssh to the box from our localhost. First, get the OTP:
MY-MAC$ vault write ssh/creds/otp_role ip=172.18.0.3
Now try to ssh using the
MY-MAC$ ssh ubuntu@localhost -p <random port>
Welcome to Ubuntu 14.04 LTS (GNU/Linux 4.4.0-101-generic x86_64)* Documentation: https://help.ubuntu.com/The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
Nice! We’ve got OTP working with
ubuntuusing Vault roles/creds and the
vault-ssh-helper! Pretty cool, but even cooler would be using Vault as a certificate authority to store and distribute signed ssh keys for individual users. Stay tuned, we’ll explore that in the next iteration of my Vault journey!