Using Vault Agent with Docker compose

Published in

Geek Culture

7 min readJul 1, 2021

Thanks to Kseniia Ryuma for the Vault Agent Caching section

I find Docker compose to be a very useful tool for test and demo purposes of local application stacks. For a recent project, I could not find good examples of using docker compose with Vault agent so I decided to write and share one here. In this post we will see how Vault Agent can be run in Docker compose to write templated html files for Nginx with Database and static passwords

We will see how all of these tools can work together

I am assuming the reader is already familiar with HashiCorp Vault and docker compose. As pre-requisites, please ensure that Docker, Docker compose, the Vault CLI (install vault) and jq (download jq)is installed.

The repo for this demo project is: kawsark/vault-agent-docker. The root directory contains a docker-compose.yml file which brings up the following services: Vault server (Dev mode), nginx, Vault agent, and Postgres. Please clone the repo and run docker compose up to bring up the above services.

git clone https://gitlab.com/kawsark/vault-agent-docker
cd vault-agent-docker
docker-compose up

Vault can now be accessed on http://localhost:8200. The root token for Vault is “root” (without the quotes). The nginx application can be accessed on http://localhost:8080, it will display the default nginx webpage until we perform the configuration steps 1 and 2 as described below.

1. Vault: configure policy and Secrets Engines

In another terminal, please run the following commands to create a policy for the nginx application, and configure two Secrets Engines. If you want to take a shortcut, just execute the ./00-secrets.sh script from repository root.

# Please ensure you are in the repo root directory
# Run ./scripts/00-secrets.sh OR use the commands below# Setup VAULT_ADDR and VAULT_TOKEN
export VAULT_ADDR=http://localhost:8200
export VAULT_TOKEN=root# Write a Policy:
vault policy write nginx nginx.hcl# Enable the kv Secrets engine and store a secret
vault secrets enable -version=2 kv
vault kv put kv/nginx app=nginx username=nginx password=sup4s3cr3t# Enable the postgres Secrets Engine 
vault secrets enable -path=postgres databasevault write postgres/config/products \
    plugin_name=postgresql-database-plugin \
    allowed_roles="*" \
    connection_url="postgresql://{{username}}:{{password}}@db:5432?sslmode=disable" \
    username="postgres" \
    password="password"# Create a Role for nginx
vault write postgres/roles/nginx \
  db_name=products \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
  GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\"" \
  default_ttl="30s" \
  max_ttl="24h"

If the above setup was successful, you should be able to read both static and dynamic secrets as shown below.

# Test reading a static secret
vault kv get kv/nginx/static
====== Metadata ======
Key              Value
---              -----
created_time     2021-06-30T14:48:54.9808961Z
deletion_time    n/a
destroyed        false
version          1====== Data ======
Key         Value
---         -----
app         nginx
password    sup4s3cr3t
username    nginx
# Test reading a dynamic secrets Database secret
vault read postgres/creds/nginx
Key                Value
---                -----
lease_id           postgres/creds/nginx/jLb7anWBjAWkIdYxYKdsDTGG
lease_duration     30s
lease_renewable    true
password           T1MboM1OBjZ7nz-5qPjD
username           v-token-nginx-yxbrxy61ShDeZXph7GgU-1625064446

2. Vault: configure an authentication method

Please run the following commands to setup AppRole Auth method from the repository root. It will mount the Auth Method, create a Role for the nginx application, then export the Role ID and Secret ID for nginx to use. If you want to take a shortcut, just execute the ./02-approle.sh script from repository root.

# Please ensure you are in the repo root directory
# Run ./scripts/01-approle.sh OR use the commands below# Enable AppRole and create a role:
vault auth enable approle
vault write auth/approle/role/nginx token_policies="nginx"# Write out a Role ID and Secret ID
vault read -format=json auth/approle/role/nginx/role-id \
  | jq -r '.data.role_id' > vault-agent/nginx-role_idvault write -format=json -f auth/approle/role/nginx/secret-id \
  | jq -r '.data.secret_id' > vault-agent/nginx-secret_id# Restart the vault-agent-demo container
docker restart vault-agent-demo

Access http://localhost:8080 on your browser, and you should be able to see the nginx application display a dynamic Postgres database credential provided by Vault as shown below. Also try accessing http://localhost:8080/kv.html to see example static secret values.

The remaining steps show the Vault agent and Docker-compose configuration elements that were used for the above demo.

2. Vault-agent: Write the configuration file

In this case we have already written an example Vault agent configuration file for your review: vault-agent/nginx-vault-agent.hcl file. It specifies how to authenticate the nginx container using AppRole in the auto-auth stanza:

auto_auth {
  method {
    type = "approle"    config = {
        role_id_file_path = "/vault-agent/nginx-role_id"
        secret_id_file_path = "/vault-agent/nginx-secret_id"
        remove_secret_id_file_after_reading = false
    }  }sink {
    type = "file"
    config = { path = "/vault-agent/token" } } 
}

It also includes two template stanzas that tell Vault Agent how to render the secrets in an application friendly way.

# Render Database secret
template {
  source = "/vault-agent/postgres.tpl"
  destination = "/usr/share/nginx/html/index.html"
}# Render KV secret
template {
  source = "/vault-agent/kv.tpl"
  destination = "/usr/share/nginx/html/kv.html"
}

Per the template stanzas above, HTML files will be rendered in the /usr/share/nginx/html directory and served by the nginx web server.

3. Vault-agent: Write a template file

We have included two template files, kv.hcl and postgres.hcl, that tell vault agent how to render secrets from a Key/Value and Database Secrets Engine respectively. Below are the main things to be aware of:

Start with the secret path at the top, for example: {{ with secret "postgres/creds/nginx" -}} for Database secrets.
Then access secrets using .Data.<keyword> format. For example, {{ .Data.password }} . For Key/Value Secrets engine version 2, you need to use the keyword .Data.data since the secrets are actually stored under the path: <kv-mount>/data . For example: {{ .Data.data.password }}.
If you are not sure of the keywords, check what is returned by using Vault CLI. For example: vault read postgres/creds/nginx or vault read kv/data/nginx/static.

4. Docker-compose: Mount a shared volume with the application

Shared volumes with Docker-compose allow us to ensure that the vault-agent can render secrets in a common directory that can be accessed by the application. For this demo, the local directory ./nginx is mounted for both vault-agent and nginx as shown below

volumes:
      - ./nginx:/usr/share/nginx/html

Below is a diagram showing the full workflow. Any updates published in the shared directory is immediately picked up by nginx.

(Optional) Testing Vault Agent Caching

Vault agent can also provide a local proxy and caching function for an application. The nginx-vault-agent.hcl file includes a listener and a cache block that enables this functionality. The listener is started on port 8200 which is mapped to localhost port 18200 in the docker compose file. In another terminal, issue the following commands to test Vault Agent caching behavior.

unset VAULT_TOKEN
export VAULT_ADDR=http://localhost:18200
vault kv get kv/nginx/static
vault read postgres/creds/nginx# Run the command a few times
vault read postgres/creds/nginx
vault read postgres/creds/nginx

There are a few interesting items to highlight:

Authentication: We did not need to set the VAULT_TOKEN environment variable because we specified use_auto_auth_token = true in the nginx-vault-agent.hcl file.
Cache: Vault agent keeps the Database lease renewed. So if you re-run vault read postgres/creds/nginx , Vault agent will keep returning the same credentials until the max TTL is reached, or if the cache is explicitly cleared. Issue docker logs vault-agent-demo to see renewal and cache response entries:

docker logs vault-agent-demo
2021-07-13T14:41:34.336Z [DEBUG] cache.leasecache: returning cached response: path=/v1/postgres/creds/nginx
2021-07-13T14:41:46.812Z [DEBUG] cache.leasecache: secret renewed: path=/v1/postgres/creds/nginx

Cache Eviction: Use the commands below to clear the cache using the cache-clear API endpoint.

# Setup environment to work with VAULT_AGENT
unset VAULT_TOKEN
export VAULT_ADDR=http://localhost:18200# Read database creds
vault read postgres/creds/nginx
Key                Value
---                -----
lease_id           postgres/creds/nginx/cN8VAqQg9xVAFNQBF4LNHWM5
lease_duration     30s
lease_renewable    true
password           ZUjZAa8n7R7hR41-X9aC
username           v-approle-nginx-zUR1sxuxg2a03wzgzvxb-1626183920# Copy the lease id from above
curl --request POST \
       --data '{ "type": "lease", "value": "postgres/creds/nginx/cN8VAqQg9xVAFNQBF4LNHWM5" }' \
       $VAULT_ADDR/agent/v1/cache-clear# Re-issue the read command to see different credentials
vault read postgres/creds/nginx
Key                Value
---                -----
lease_id           postgres/creds/nginx/whRBAl2gCRprtWNxILIrw2Dr
lease_duration     30s
lease_renewable    true
password           -5fx1FiT0tRSISzAUOZF
username           v-approle-nginx-vW6CId8X99LUkurp7Klf-1626187012

Cleanup

To cleanup, run the ./scripts/cleanup.sh script, or use the commands below.

# Please ensure you are in the repo root directory
# Run ./scripts/cleanup.sh OR use the commands belowdocker-compose down
rm -f ./vault-agent/*role_id
rm -f ./vault-agent/*secret_id
rm -f ./vault-agent/token
rm -f ./nginx/index.html
rm -f ./nginx/kv.html

Summary

Vault agent can be used to delegate Authentication and Secrets retrieval tasks on behalf of an application. This post provides the various configuration bits needed to do this with Docker Compose. I hope this is a starting point for you to build cool projects showing Vault integration with Docker. Please leave any feedback below.