Set up Vault with Vagrant and Ansible

Photo by Artem Kovalev on Unsplash

In the DevOps space, we take for granted that we can now easily scale our infrastructure outside of the boundaries of our safe and secured data centers, spin up as many instances that we could possibly want, and all with a touch of a button (or maybe a few keystrokes). But with great power comes great responsibility (thanks uncle Ben!). Are we managing our secrets and sensitive data in a way that we can tightly control access and audit who has accessed it? We have all been guilty using the one SSH key on multiple machines or even saving that long string token/password in a text file on your local machine to copy paste for later use. But what if your Azure or AWS credentials got accidentally commit in plain text to your GitHub repo and now become a free service for running bitcoins? How do we reduce the blast radius of the impact of such an incident?

HashiCorp Vault is becoming a very popular open source secret management solution that will securely store your secrets (password, database, cloud access keys and any other key value you want) and can be accessed once authenticated, an example of this is providing dynamic AWS credentials or even one-time SSH passwords. Using dynamic ephemeral credentials reduces the risk of credentials that are logged to disk or otherwise exposed because they will be rotated automatically or have a short time to live. Vault also offers encryption as a service which fortifies your data at rest and in transit, and PKI as a service allowing you to dynamically generate signed TLS certificates on demand. Access to the data in Vault is tightly controlled, with many authentication methods available such as LDAP, GitHub, Username/Password, token etc.

This guide will help you get up and running with a local Dev environment using Vagrant and Ansible and we will get our hands dirty with an exercise to create an SSH one-time password issued by Vault to log into a server.

What are we going to build:

1. A Vault server — in Dev mode
2. A Consul server — This will be our backend storage and works natively with Vault.
3. Client — Ubuntu server
4. Server — Ubuntu server, we will use this to demonstrate Vault provisioning a one-time Password.

What you are going to need.

1. Vagrant installed — Get it here
2. Virtualbox installed — Get it here
3. Ansible — if you are using Mac you can run `brew install ansible` if you have homebrew installed.
4. Clone of this Git Repo

Let’s take a look at what we have in the cloned directory.

Vagrant file

Our Vagrant file declares what infrastructure we want to provision. We are going use ansible to perform the installation and configuration of Vault and Consul on our “Vault server” and set up our “client” server and “OTP” server.

The servers we are going to provision are:

Vault/Consul server — IP address 192.168.50.100 and using ansible to provision it using the install.yml file under the vault directory.

Client — IP address of 192.1268.50.101 and using the install.yml file under the client directory.

OTP server — IP address of 192.1268.50.102 and using the install.yml file under the OTP directory.

The Ansible files

The Vault/Consul Server (install.yml)

The install.yml file used by ansible will:

• Create a Vault User
• Download Consul and configure it as a service in -dev mode
• Download Vault and configure it as a service in -dev mode using Consul as it’s backend.
• Unseal Vault and save the root token locally under the /rootKey folder

The Client (install.yml)

The install.yml file is used by ansible to download Vault and use it to access the Vault Server

The SSH-OTP Server (install.yml)

This install.yml file is used by ansible to download and configure the Vault SSH Helper to allow the server to consume the One-Time Password created by Vault.

Now let’s get our environment up and running. To do this run vagrant up and provision your 3 servers.

Note if you have issues with ansible creating files to store the vault keys then you can run sudo vagrant up.

Bringing machine ‘vault’ up with ‘virtualbox’ provider…
Bringing machine ‘client’ up with ‘virtualbox’ provider…
Bringing machine ‘otp’ up with ‘virtualbox’ provider..
.....
TASK [Download Consul] *********************************************************
changed: [vault]
TASK [Unzip Consul] ************************************************************
changed: [vault]
TASK [Make Consul a service] ***************************************************
changed: [vault]
TASK [Ensure config directory exists] ******************************************
changed: [vault]
........
TASK [Write unseal keys to files] **********************************************
changed: [vault -> localhost] => (item=[0, ‘3f6299696c198070b0c0d8fd935705f0e53a4bd95a587a55705abc0846b405b32f’])
changed: [vault -> localhost] => (item=[1, ‘e85f00e1103cc2c0aa309aa08d6e139bca7baf44bf2842df489d4dbc40f11a1c83’])
changed: [vault -> localhost] => (item=[2, ‘d23f09eaee91da83ed5491743f7113c1f0954e0dbe23a582f9601cb2ea7f394df6’])
changed: [vault -> localhost] => (item=[3, ‘0ace41f5005a7d7a49371400c8cff264b4e26ab50ba04433346558e99102023d6b’])
changed: [vault -> localhost] => (item=[4, ‘88ba279b850996ac6398878d48c86f690a7afce625f20110e873153b6cec1f55d5’])

Now that our infrastructure is up and running we can SSH into our Vault server by running vagrant ssh vault (or if you need to use sudo, then sudo vagrant ssh vault).

Set your environment variable so the server knows where to find Vault.

export VAULT_ADDR=’http://127.0.0.1:8200'

Login into Vault with your token found in /vault/rootKey/rootKey.txt(in production you would not use the root key!)

Vault login s.a0fJ1fYMwvFGbFaSoqf3P6E0 (remember to use your token)
Key Value
— — — — -
token s.a0fJ1fYMwvFGbFaSoqf3P6E0
token_accessor JR3OMCNehciCWp74VDDXdjwl
token_duration ∞
token_renewable false
token_policies [“root”]
identity_policies []
policies [“root”]

And now you're in, so let’s take Vault for a test drive.

One Time SSH Password

A great use case for Vault is the One-Time SSH Password (OTP) SSH secrets engine that will allow Vault to issue a One-Time Password every time a client wants to SSH into a remote host using a helper command on the remote host to perform verification. This solves a few problems, ensuring that we are restricting access for anyone to SSH into our servers and make config changes (keeping with the DevOps immutable ethos), and limiting the “one pem key to rule them all” scenario. The authentication flow will look like this:

1. From the “client” server we are authenticating to vault with a token and request an SSH one-time password.
2. Vault will check our access and then provide us password to use to ssh to our “otp” server.
3. From the “client” server we will ssh into the server with our one-time password.
4. The Vault SSH agent will check with vault to see if the password is valid and allow us to use it. Once used it can not be used again.

First, we need to set up the SSH backend on the Vault server (logged in to the vault server)

vault secrets enable ssh

Now create an otp role with a key type as OTP and the default ssh username as vagrant.

vault write ssh/roles/otp_key_role key_type=otp default_user=vagrant cidr_list=0.0.0.0/0

We now need to set up a method for our “client” to authenticate to Vault. We are going to use the User/Password method.

vault auth enable userpass

Then we create a policy to allow for a one-time password :

vault policy write admin -<<EOF
path “sys/mounts” {
capabilities = [“list”,”read”]
}
path “ssh/*” {
capabilities = [“read”,”list”,”create”,”update”]
}
path “ssh/creds/otp_key_role” {
capabilities = [ “update” ]
}
EOF

Time to create a new user that client server will log in as. We are going to create “admin” as the user who will use the OTP.

vault write auth/userpass/users/vault password=vault policies=admin

Now open up a new terminal window and go to the cloned directory and turn vagrant ssh client or sudo vagrant ssh client. Login as the new user you created.

vault login -method=userpass username=vault password=vault

Request a one-time password from Vault with the “otp” server IP address.

vault write ssh/creds/otp_key_role ip=192.168.50.102
Key Value
— — — — -
lease_id ssh/creds/otp_key_role/u9AuTq4m5i4hcu6fL5XqhOQI
lease_duration 768h
lease_renewable false
ip 192.168.50.102
key 0f79a06c-3458–53c3–1282–0be992f3560b
key_type otp
port 22
username vagrant

Now issue the following command

ssh vagrant@192.168.50.102
The authenticity of host ‘192.168.50.102 (192.168.50.102)’ can’t be established.
ECDSA key fingerprint is SHA256:G1J5ek/T6KuCCT7Xp2IN1LUslRt24mhmhKUo/kWWVrs.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘192.168.50.102’ (ECDSA) to the list of known hosts.
Password:

Now paste the key from earlier as the password. The SSH connection should successfully log you in. Any further attempts will fail, as the password is good for one use only!

Once you have finished you can tear down your environment with a simple command

vagrant destroy -f or sudo vagrant destroy -f

Conclusion

HashiCorp Vault is a great answer to the problem of secrets management in a very elegant way and I’ve seen many clients that could benefit from the centralization, dynamic provisioning and rich auditing that Vault provides. All of Vault’s capabilities can also be accessed via the HTTP API as well as CLI and UI, which comes in handy when access use Vault to store our passwords and access them in our CI/CD pipeline.

I hope this helps you get started using Vault!

PS: If you liked the article, please support it with claps. Cheers