Setup Hashicorp Vault Using Ansible

Oct 1, 2018 · 7 min read
“white wooden door surrounded by plants” by Daniel von Appen on Unsplash

Every application needs secret management which helps with authorization or authentication to the system using username password, database credentials, API tokens, TLS certificate, cloud provider access keys, and many other secrets. Developers used to share them through traditional methods like email, push them to the source code, keep them in configuration management like ansible, store them somewhere like S3 which creates secret sprawl. These methods have many flaws which prevent us from knowing who really have access to all these secrets, who accessed them in the past, the location of secret, how to rotate these secrets. We don’t want the whole world to know about these secrets. We want to keep them safe. If we want to expire secrets, we need to manually perform actions, create a new secret, distribute them and use them. We do not have any audit logs to track who requested a particular secret.

HashiCorp Vault is a popular open source tool for secret management that allows users to store, manage and control access to tokens, username password, database credentials, TLS certificate, and many other secrets. Vault is used to manage secrets securely in a central location which prevents secret sprawl. Vault provides strict fine-grained control over who can access secrets. Value provide audit policies and audit logs on secrets which provide the strong guarantee and strong visibility into secret access. Vault handles leasing, key revocation, key rolling, and auditing. Vault supports pluggable mechanisms known as secrets engines for managing different secret types.

Applications can leak credentials through logs, diagnostic output like exception trace, shipping to an external monitoring tool etc. Vault provides dynamic secrets which help provide short-lived ephemeral credentials to applications instead of long live credentials. With dynamic secrets, even if an application writes it to an external system, it is valid for a specific period of time. We have different credentials for different services as they are dynamically fetched and unique, which help us identify exactly from where credential is used. This helps us in doing proper revocation, as we know what exact credential is compromised using audit logs. This prevents to reduce blast radius on credential compromise as no other service is effected when a specific credential is revoked.

Vault provides encryption as service. Vault shield encryption implementation from the end user and does key management instead of relying on the developer to do encryption using keys. This helps in ensuring cryptography is done in right way and key management with lifecycle management of keys.

Let’s list some of the benefits of using Vault:

  1. Prevents secret sprawl
  2. Provides dynamic secret
  3. Encryption as a service
  4. Provide audit logs
  5. Revocation is easy

Let’s look in the setup of the vault using Ansible. We are going to create an Ansible Role for Vault setup so we can reuse it. We will begin by creating a new user account named “vault” which will help with a secure setup. We will use this account to isolate the ownership of vault. We don’t create any home directory or shell for this user so that user can’t log in to a server.

- name: Creating vault user group
name: "{{ vault_group }}"
become: true
- name: Creating vault user
name: "{{ vault_user }}"
group: "{{ vault_group }}"
system: yes
shell: "/sbin/nologin"
comment: "vault nologin User"
createhome: "no"
state: present

Next, we need to download vault archive from here on our remote vault instance. This will give a zip archive file. To unzip vault archive, we need to install unzip so we can unzip vault archive and takeout needed binary. Once this is done, we need to unzip vault archive, move our vault binary to “/usr/local/bin” and make vault user as the owner of this binary with needed permissions.

- name: Install prerequisites
name: "{{ item }}"
update_cache: yes
with_items: "{{ vault_install_prerequisites }}"
become: yes
- name: Download binary
dest: /tmp/vault_{{vault_version}}
owner: "{{ vault_user }}"
group: "{{ vault_group }}"
mode: 0755
checksum: "{{vault_checksum}}"
register: vault_download
- name: "Unzip vault archive"
src: "{{ vault_download.dest }}"
dest: /usr/local/bin
copy: no
owner: "{{ vault_user }}"
group: "{{ vault_group }}"
mode: 0755

We need to set binary capabilities on Linux, to give the Vault executable the ability to use the mlock syscall without running the process as root.

- name: "Set vault binary capabilities"
path: /usr/local/bin/vault
capability: cap_ipc_lock+ep
state: present

We need to setup systemd init file to manage the persistent vault daemon.

Description=Tool for managing secrets
ExecStart=/usr/local/bin/vault server -config=/etc/vault.hcl
ExecReload=/usr/local/bin/kill --signal HUP $MAINPID
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK

We need to set above content into Systemd service file. Finally, start the vault server.

- name: Copy systemd init file
src: init.service.j2
dest: /etc/systemd/system/vault.service
owner: root
group: root
notify: systemd_reload
- name: config file
src: vault.hcl.j2
dest: "{{ vault_config_path }}"
owner: "{{ vault_user }}"
group: "{{ vault_group }}"
- name: vault service
name: vault
state: started
enabled: yes

Vault starts in an uninitialised state. We need to set up initial parameters to unseal so it can be used. We need to set an environment variable so it is not needed to be provided on running vault commands. We are running vault in HTTP mode as being used by internal services. We can add TLS related changes by providing the certificate details in “vault.hcl” config file.

export VAULT_ADDR=

First we are going to do complete process manually before doing it with Ansible for better understanding. We need to run init command “vault operator init” and get a response as the root token and unseal keys.

Once this is done, vault becomes initialised but remains seal. In seal state, it can’t store or provide any secret, more like locked unusable state. We need to unseal it with the required number of unseal keys. To provide keys we need to run command “vault operator unseal” and provide a unseal key.

Keep running this command, till we unseal vault successfully. In the default case, there are 5 unseal keys generated and we need to provide 3 of them.

Once this unseal is done, we can check “vault status” to make sure it is unsealed.

Now lets do all this using ansible so we don’t need to do anything manually. Most important part is to make sure we have our unseal keys and root key secure as they are needed to unseal vault and do admin tasks. First, we are going to create unseal key and root key folder where we store our keys. We are going to create keys using “vault operation init” and register it in a variable. These keys then stored in our files so we can access them later.

- name: Create unseal directories
path: "{{ unseal_keys_dir_output }}"
state: directory
delegate_to: localhost
- name: Create root key directories
path: "{{ root_token_dir_output }}"
state: directory
delegate_to: localhost
- name: Initialise Vault operator
shell: vault operator init -key-shares=5 -key-threshold=3 -format json
register: vault_init_results
- name: Parse output of vault init
vault_init_parsed: "{{ vault_init_results.stdout | from_json }}"
- name: Write unseal keys to files
dest: "{{ unseal_keys_dir_output }}/unseal_key_{{ item.0 }}"
content: "{{ item.1 }}"
with_indexed_items: "{{ vault_init_parsed.unseal_keys_hex }}"
delegate_to: localhost
- name: Write root token to file
content: "{{ vault_init_parsed.root_token }}"
dest: "{{root_token_dir_output}}/rootkey"
delegate_to: localhost

With this we have our vault init with unseal keys and root keys stored in local directory which can be used to unseal vault. We are going to divide this in two separate ansible roles vault-init and vault-unseal as we might need to unseal multiple times when vault goes down. To unseal, we need to read unseal keys and pass them during “vault operator unseal” command. Once this is done, we have our vault unsealed with ansible.

- name: Reading unseal key contents
command: cat {{item}}
register: unseal_keys
with_fileglob: "{{ unseal_keys_dir_output }}/*"
delegate_to: localhost
become: no
- name: Unseal vault with unseal keys
shell: |
vault operator unseal {{ item.stdout }}
with_items: "{{unseal_keys.results}}"

Now vault is ready for handling any secret management. We need to use root key for accessing vault which we got during the init process. Root token has root permissions to our vault deployment. We will set token as environment variable so we do not need to specify in each command.

export VAULT_TOKEN=acd462e7-f8da-6554-6774-04c8b91aa520

We can test by saving a secret in the vault and then fetching the same secret to confirm that it is working as expected. We are writing the value as “secretData” for “secret/content”.

vault write secret/content value=secretDataSuccess! Data written to: secret/message

We read the value of “secret/content” from the vault and get “secretData” back which was stored previously. This confirms that it is working as expected.

vault read secret/contentKey                 Value---                 -----refresh_interval    768hvalue               secretData

When vault service stops due to some reason we need to again unseal it with same unseal keys, so making sure we keep these unseal keys as safe as possible.

The complete code can be found in this git repository:

Click here for “Introduction To Ansible

Click here for “Introduction To Ansible Roles

PS: If you liked the article, please support it with claps. Cheers

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade