How to Set Up Hashicorp Vault Secrets Server in AWS

Published in

Tulco Labs

9 min readJul 2, 2020

Managing credentials in software projects can be tough. You never want to expose sensitive data unintentionally. Anyone can make this mistake. Even large companies such as Facebook have made the mistake of storing secrets in an insecure way. This is because storing secrets securely creates competing technical challenges — On one hand, you want to maintain proper access control, you want to keep good audit logs of who accessed resources, and you want your secrets to remain confidential (they are a secret after all!). On the other hand, you need your developer workflow to be as efficient as possible. And there is the rub.

In the past, we tried storing an encrypted file containing our secrets in source control using ansible-vault. However, this was not an ideal configuration. Access to the encrypted file could only be controlled at the level of the repository, and any time developers needed to run a program, they would need to un-encrypt the file on their local machines, meaning we couldn’t protect the secret from developers. Additionally, it wasn’t possible to properly audit access to the file. These encrypted files also appeared meaningless in git diff comparisons.

There are many different tools for secrets management, but today I will be covering how to secure secrets using Hashicorp Vault. Vault is an open-source project that provides a secure interface to access secrets for a variety of applications.

I prefer Vault to other alternatives for a few reasons:

First, it’s open-source which means it can be deployed in a variety of scenarios.
Second, it can interface with many different programming languages, and also supports an API.
Lastly, it supports all the security and audit functions I am looking for in a secrets store. Secrets remain secret throughout the entire developer workflow.

In the sections that follow, I will be covering how to set up a Vault secrets server on an AWS EC2 instance using AWS S3 as a storage backend.

Notes before beginning

This tutorial will use resources that may incur a cost on your AWS bill. Please refer to the AWS pricing to see how your bill may be impacted. Generally, the resources used here will cost fewer than $10 a month

Setting up the s3 bucket

First, we will create the s3 bucket to use as storage for our Vault instance

Navigate to the s3 console from the AWS management console
Click the create bucket button
Give it a name. Leave the other settings as they are
Remember the bucket ARN for later
You can find the bucket ARN by selecting the bucket, then going to Permissions → Bucket Policy

Setting up the IAM policies and Roles

We will be using an IAM user and role to authenticate with Vault once it is deployed.

Set up the IAM Policy

Navigate to the IAM console from the AWS management console
Click policies on the side tab
Click create policy
Select the visual editor for creating the policy
Choose s3 as the service
In the list section, select ListBucket
In the read section, select GetObject
In the write section, select PutObject
In the Resources section: For Bucket enter in the ARN for the bucket you created in the previous section
Also in Resources section: For Object enter in the ARN for the bucket you created in the previous section for the bucket option, select any for the Object option
Click Review Policy
Give the policy a name and a description
Click create policy

The JSON of the policy should look like this at the end

{     
    "Version": "2012-10-17",     
    "Statement": [         
        {             
            "Sid": "VisualEditor0",             
            "Effect": "Allow",             
            "Action": [                 
                "s3:PutObject",                 
                "s3:GetObject",                 
                "s3:ListBucket"             
             ],             
            "Resource": [                 
                "<BUCKET_ARN>/*",                 
                "<BUCKET_ARN>"             
            ]         
        }     
    ] 
}

Create IAM role

We will now create an IAM role that will allow an ec2 instance to connect to this bucket

Click on Roles on the side tab
Click Create Role
Select EC2 as the use case
Attach the policy you just created to the role
Click next until you get to the Review step
Give your role a name and a description
Click Create Role

Create IAM user

Lastly, we will create an IAM user and assign the policy to it. We will generate access keys for this user when initializing Vault

Click on Users on the side tab
Click add user
Give the user a name
Select programmatic access for the access type
Click Next: Permissions
Attach the s3 policy from the previous section to the user
Click next until you get to the Review step
Review the information and click create user
Save/download the user credentials on the next screen
Create another policy that will allow this iam user to authenticate with the AWS sdk while using Vault. Navigate to the permissions tab and click Add Permissions
Select add existing policies directly
Click create policy
Put in the following policy definition in the json tab:

{     
    "Version": "2012-10-17",     
    "Statement": [         
        {             
            "Effect": "Allow",             
            "Action": [                 
                "ec2:DescribeInstances",                    
                "iam:GetInstanceProfile",                 
                "iam:GetUser",                 
                "iam:GetRole"             
            ],             
            "Resource": "*"         
        },         
        {             
            "Effect": "Allow",             
            "Action": [                 
                "sts:AssumeRole"             
            ],             
            "Resource": [                 
                "<USER_ARN>"             
            ]         
        }     
    ] 
}

Click Review Policy
Give your policy a name and description
Click create policy
Attach this policy to the new user just created

Setting up the ec2 instance

We will now launch the ec2 instance that will host our vault server

Navigate to the ec2 console from the AWS management console
Select Instances from the right side tab
Click launch instance
Select the AWS Linux 2 ami (the topmost option)
Select an instance class, for this tutorial we will use a nano size instance
Click Next: Configure Instance Details
Under IAM role, select the IAM role we created in the previous step
Click Next: Add Storage
The defaults are fine. Click Next: Add Tags
Give your instance a Name tag so you can identify it after launch
Click Next: Configure Security Group
Create a new security group
Give your security group a name and a description
Add a new TCP rule with port range 8200. For source choose My IP

Note: In production you should only limit access to the security groups of the services that require access to Vault

Click Review and Launch
Click Launch
If you have an existing key pair you would like to use select it, otherwise download a new key pair
Last click Launch Instance then View Instance

Installing Vault

Next, we will install vault onto the instance.

If you are not already ssh’ed into the machine, ssh now using the key pair generated in the launch ec2 step
If this is a new key you may get a permission error when you attempt to ssh in. If this happens modify the permissions on the key and try again

chmod 400 <path/to/key.pem>

update the instance

sudo yum update

Install vault (find other versions/binaries on the releases page)

wget https://releases.hashicorp.com/vault/1.4.2/vault_1.4.2_linux_amd64.zip

Unzip it

unzip vault_1.4.2_linux_amd64.zip

Move the binary to a location on $PATH

sudo mv vault /usr/local/bin/vault

Verify that vault installed properly

vault version

Create a privileged user to run vault

sudo useradd --system --home /etc/vault.d --shell /bin/false vault

Generating a self-signed ssl certificate

A ssl certificate is necessary to ensure secrets remain secure during transit. Vault is configured to run with an https connection and will not work as intended without proper https communication. If you have an ssl certificate to use you can skip this step.

Note: steps taken from https://www.akadia.com/services/ssh_test_certificate.html

ssh into the ec2 machine created in the previous step

If this is a new key you may get a permission error when you attempt to ssh in. If this happens modify the permissions on the key and try again

chmod 400 <path/to/key.pem>

Create a directory for ssl and change directory into it

sudo mkdir /etc/vault.d/.ssl & cd /etc/vault.d/.ssl

Generate a private key. Remember the password you use

openssl genrsa -des3 -out server.key 1024

Generate a certificate signing request (CSR). You will be asked for the password of server.key from step 3. This will prompt for some details, skip the part when it asks for a challenge password

openssl req -new -key server.key -out server.csr

Remove the passphrase from the key

cp server.key server.key.org 
openssl rsa -in server.key.org -out server.key

Generate the certificate

openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

Discard the temporary key file and return to home directory

rm server.key.org && cd

Install Vault as a Service

Next, we will install Vault as a service to ensure it is available after a server reboot. We will use systemd for this.

Create a vault service file

sudo touch /etc/systemd/system/vault.service

Add the following to the service file

[Unit] 
Description="HashiCorp Vault - A tool for managing secrets" 
Documentation=https://www.vaultproject.io/docs/ 
Requires=network-online.target 
After=network-online.target 
ConditionFileNotEmpty=/etc/vault.d/vault-config.hcl 
StartLimitIntervalSec=60 
StartLimitBurst=3[Service] 
User=vault 
Group=vault 
ProtectSystem=full 
ProtectHome=read-only 
PrivateTmp=yes 
PrivateDevices=yes 
SecureBits=keep-caps 
AmbientCapabilities=CAP_IPC_LOCK 
Capabilities=CAP_IPC_LOCK+ep 
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK 
NoNewPrivileges=yes 
ExecStart=/usr/local/bin/vault server -config=/etc/vault.d/vault-config.hcl 
ExecReload=/bin/kill --signal HUP $MAINPID 
KillMode=process 
KillSignal=SIGINT 
Restart=on-failure 
RestartSec=5 
TimeoutStopSec=30 
StartLimitInterval=60 
StartLimitIntervalSec=60 
StartLimitBurst=3 
LimitNOFILE=65536 
LimitMEMLOCK=infinity[Install] 
WantedBy=multi-user.target

Create a directory for Vault configuration file

sudo mkdir --parents /etc/vault.d

Create the vault configuration file (I use vim here but feel free to use the text editor of your choice)

sudo vi /etc/vault.d/vault-config.hcl

Define the configuration

listener "tcp" {     
    address = "0.0.0.0:8200"     
    tls_cert_file="/etc/vault.d/.ssl/server.crt"     
    tls_key_file="/etc/vault.d/.ssl/server.key" 
}backend "s3" {     
    bucket = "<your_bucket_name>"     
    region = "<bucket_region>" 
}disable_mlock=true

Give ownership of the vault folder to the vault user

sudo chown --recursive vault:vault /etc/vault.d

Change permissions on the config file

sudo chmod 640 /etc/vault.d/vault-config.hcl

Enable and start the service. Check the status of the service once it has been started

sudo systemctl enable vault 
sudo systemctl start vault 
sudo systemctl status vault

Note: If you have an issue with starting the service here are some tips for debugging

At this point you should have vault up and running. You will likely see a warning about your ssl certificate being invalid as it is self-signed and does not match the domain name that Vault is running on. In production you will want to generate a proper ssl certificate to use for Vault. For now you can run the following command to ignore the error for the rest of the tutorial.

export VAULT_SKIP_VERIFY=true

You will need to initialize the vault server. This prepares vault to receive data and only needs to be run once. After this command is run you will see your unseal keys and root token key. These are the keys to the kingdom and need to be saved securely! If you lose this information it will require you to delete and re-install vault.

vault operator init

Before you can do any meaningful operations on the Vault server, you must unseal it. You will need to run this command multiple times, each time provided a correct unseal key generated in step 10 (by default 3, a majority of the unseal keys) before the vault becomes unsealed. You can read more about seal/unseal here

vault operator unseal

Configure Authentication with AWS

Vault supports many different authentication methods. We will be using AWS as our authentication method

First login with the root token obtained during the init step

vault login <ROOT_TOKEN>

Create a policy document to configure permissions to vault. Put the following in the policy document

# Dev servers have version 2 of KV secrets engine mounted by default, so will 
# need these paths to grant permissions: 
path "secret/data/*" {   
    capabilities = ["create", "update"] 
}path "secret/data/foo" {   
    capabilities = ["read"] 
}

The above policy allows write operations on any path, except for the foo path which is read only

Write this policy into vault, choose any name you want for <POLICY_NAME>, use the file you created in step 2 for <POLICY_FILE>

vault policy write <POLICY_NAME> <POLICY_FILE>

Enable the aws auth method

vault auth enable aws

Write the policy to the aws iam user created in the previous steps

vault write auth/aws/role/<IAM_USER_NAME> auth_type=iam bound_iam_principal_arn=<USER_ARN> policies=<POLICY_NAME>

vault login -method=aws aws_access_key_id=<AWS_ACCESS_KEY> aws_secret_access_key=<AWS_SECRET_KEY>

Test the Policy

Next we will test the policy by attempting to write a secret

First, ensure that the key-value v2 secret engine is enabled with vault secrets list

Path          Type         Accessor              Description 
----          ----         --------              ----------- 
cubbyhole/    cubbyhole    cubbyhole_31493a22    per-token private secret storage 
identity/     identity     identity_1d137c3d     identity store 
secret/       kv           kv_3f592852           n/a 
sys/          system       system_b231fc50       system endpoints used for control, policy and debugging

If you do not see secret/ enable it with

vault secrets enable -path=secret/ kv-v2

Once you have verified that the secret/ path is enabled, try setting a secret with

vault kv put secret/cred password=”super-secure-password”

Your output should look as follows:

Key              Value 
---              ----- 
created_time     2020-06-11T15:03:20.957311824Z 
deletion_time    n/a 
destroyed        false 
version          1

Since the foo path is read only, attempting to write to foo should fail.

vault kv put secret/foo data=”this-wont-work”

Output should look as follows:

Error writing data to secret/data/foo: Error making API request.URL: PUT https://127.0.0.1:8200/v1/secret/data/foo 
Code: 403. Errors:* 1 error occurred:  
        * permission denied

Conclusion

At this point, you now have a successful running Vault server with an AWS account authorized to write to it. You could use a library such as HVAC for accessing Vault in python code, or one of the other client libraries, or access Vault via their API.