Onboarding Applications to Vault Using Terraform — a Practical Guide

Published in

HashiCorp Solutions Engineering Blog

8 min readJul 6, 2021

Originally published on the HashiCorp blog.

TL;DR: Use sensible naming standards, Infrastructure-as-Code (IaC) tool such as Terraform, ACL policy templates, and automated workflows to scale secrets management using Vault. An example guide and demo is published in the operations/onboarding folder of the vault-guides repo.

Motivation: Vault use generally starts with one use-case that mitigates a specific risk. Soon, the demand for Vault “spreads like wildfire” (real customer quote). At this point, a systematic method is needed to onboard and service a myriad of applications and use-cases. A manual approach is unsustainable, and goes against the ethos of using Vault in the first place.

A practical approach: We recommend codifying Vault configurations and using an IaC approach to manage Vault. A few key benefits of this approach include:

Repeatability: IaC allows for a predictable and consistent process for applying changes. Every configuration begins with code.
Version Control: Configuration codified in a version control system is the source of truth. Security policy and organizational standards can be enforced on IaC modules. All changes are tracked and inspected through a code review process.
Automated workflows: Properly implemented IaC practices enable GitOps or CI/CD driven workflows that do not require manual intervention. Changes and updates can be safely tested before applying to production.

Key outcomes from the above benefits include increased productivity from automation, and mitigation of risk by reducing human errors. The benefits of using IaC are amplified when onboarding applications to vault since this is a process that may be repeated 100s of times.

Methodology

In this post we will demonstrate techniques for onboarding applications using the Terraform Vault Provider. Below are some key milestones of this process.

Each step in the above diagram is described in more details below along with the corresponding demo steps. To follow along with the included demo, please install the following pre-requisites: docker (get Docker), docker compose (install Docker Compose) terraform cli (Terraform install), and git (installing git).

Set Up the Repository

Please clone the repository as below and cd into the project directory.

git clone git@github.com:hashicorp/vault-guides.git
cd vault-guides/operations/onboarding

Pleasecd into the docker-compose directory and run docker compose up as shown below. The remaining terminal snippets in this post will assume that you are in the project directory vault-guides/operations/onboarding.

cd docker-compose/ && docker compose up

Optionally, if you prefer using the tool make, there is a Makefile included in the project directory root. Run make info to see the available targets.

Bootstrap Vault

This step involves initializing + unsealing Vault, creating Vault Namespaces (for Vault Enterprise), and creating one or more Administrators. In another terminal, please use the following commands to initialize and unseal the Docker compose demo Vault instance.

# Assuming you are in the project root directory
cd docker-compose/scripts
00-init.sh
export VAULT_ADDR=http://localhost:8200
export VAULT_TOKEN=<token-from-init-script>
vault token lookup

If you need to look up the root token again, it is stored in the file: docker-compose/scripts/vault.txt. If you will be using an existing Vault cluster, please skip running 00-init.sh and instead run export VAULT_ADDR=<Vault-server-domain> to point to your Vault server.

Admin Token (Optional): You may prefer using an Admin token instead of Root, for example if using an existing cluster. If so, please create an admin token using the vault-admin.hcl policy file as shown below. This admin policy is authored based on the Vault Policies learn guide.

# Assuming that VAULT_TOKEN is set with root or higher Admin token
vault policy write learn-admin admin-policy.hcl
vault token create -policy=learn-admin
export VAULT_TOKEN=<token-from-above command>
vault token lookup

Establish naming convention

Naming standards allow applications to login and read secrets from consistent and predictable paths. Each application should have a unique identifier and be part of a logical group such as a line of business (LOB), project etc. Below are the usual set of paths to plan for, along with example values:

Auth Method Mount path — e.g. approle, k8s-us-east1
Auth Method Role name — e.g. LOB1-app100 , nginx
Secret Engine Mount path — e.g. kv/LOB1 , db/postgres-us-east1-prod
Secret access path — e.g. kv/LOB1/app100 , kv/nginx
Entity name — e.g. LOB1-app100, nginx

It is encouraged that you build this out for a representative use-case in your organization. If using Vault Enterprise Namespaces, the above paths will be pre-pended by the Namespace. E.g. the secret access path for Namespace ns1 will be ns1/kv/LOB1/app100 andns1/kv/LOB1/data/app100 for Key/Value Secrets Engine version 1 and version 2 respectively.

Terraform configurations

We will use Terraform to onboard an application named nginx. In the file variables.tf, we have declared an entitiesvariable that will hold a list of applications. To onboard more applications, we just need to append to this list . Please run the following commands to create all of the configurations.

# Please ensure you have VAULT_ADDR and VAULT_TOKEN set
cd terraform
terraform init
terraform plan
terraform apply --auto-approve# Restart vault-agent container to render secrets immediately
docker restart vault-agent

Access http://localhost:8080 on your browser, and you should be able to see the nginx application display a dynamic Postgres database credential provided by Vault as shown below. Also try accessing http://localhost:8080/kv.html to see example static secret values.

The Terraform configurations for this demo are described in more detail below along with the corresponding source file names:

Application Entity — entity.tf: Pre-creating the application entity is optional but encouraged. It allows for easier auditing and more flexibility in attaching ACL policies. Please log on to the Vault UI on http://localhost:8200 with the Root token, then click Access > Entities to see two pre-created Entities: nginx and app100. Clicking into these Entities will display an alias for AppRole Authentication method, and the mapped Entity ACL policies.

Authentication Method—auth.tf: this demo uses the AppRole Auth method, which is a type of “Trusted Orchestrator” Secure Introduction pattern. An Authentication Method Alias links the Entity to the AppRole Role.

ACL Policy — entity.tf: We recommend using templated policies to reduce the overhead of policy management. This demo uses two templated policies: kv_rw_policy for accessing Key/Value secrets, and postgres_creds_policy for accessing dynamic Postgres credentials.

These elements are represented as a Terraform graph snippet below.

Terraform graph snippet for Authentication, Entity and ACL Policy

Both ACL policies above have rules based on the {{identity.entity.name}} parameter. For example, if the entity name is nginx, during runtime, the effective path will be: kv/data/nginx/* for KV, and postgres/creds/nginx for Database secrets engine (see Templating Parameters for more details).

Secrets Engine- kv-v2.tf and database.tf: Relevant Secrets Engine should be mounted, and application specific roles should be created as part of onboarding. These are shown in a similar graph snippet as before.

Terraform graph snippet for Secret Engine mount and roles

Application integration

Now that the Vault configurations are created, we need the application to Login to Vault using AppRole credentials and fetch a secret. The demo uses Vault Agent to achieve this (see App Integration for more patterns).

The file nginx-vault-agent.hcl specifies how to authenticate the nginx container using AppRole. It also links two template files, kv.hcl and postgres.hcl, that tell vault agent how to render secrets from a Key/Value and Database Secrets Engine respectively.

Onboarding the next application

To onboard another application, simply add its name to default value of the entities variable in variables.tf as shown below for app200 .

# Snippet from variables.tf after adding app200
variable "entities" {
    description = "A set of vault clients to create"
    default = [
        "nginx",
        "app100",
        "app200"
    ]
}

Then run terraform apply to create the additional Vault configurations for this application.

# Ensure that VAULT_TOKEN was set from before
cd terraform
terraform apply --auto-approve

Verify from the Vault UI that there is a new entity called app200 , with an alias to the AppRole Auth method:

Vault screenshot showing a new app200 entity being added

A new Role ID and Secret ID has also been created which can be obtained by running the terraform output command. We can use this to test authentication and secret access as shown below. Note that the Role ID, Secret IDs and Vault Token will be unique in your case.

# Get Approle creds
terraform output role_ids | grep app200
  "app200" = "ff2795c9-7f42-4233-eace-075c28869199"
terraform output secret_ids | grep app200
  "app200" = "e76ed3e5-e2c5-fd9e-14f1-554b40674d54"# Login using AppRole
vault write auth/approle/login \
  role_id=ff2795c9-7f42-4233-eace-075c28869199 \
  secret_id=e76ed3e5-e2c5-fd9e-14f1-554b40674d54
Key                     Value
---                     -----
token                   s.JGHR7HEc0I51kOvYSfR1SfGj
token_accessor          RgOTOtM2InSfF8oz62tvOOZF
token_duration          768h
token_renewable         true
token_policies          ["default" "kv_rw_policy" "postgres_creds_policy"]
identity_policies       ["kv_rw_policy" "postgres_creds_policy"]
policies                ["default" "kv_rw_policy" "postgres_creds_policy"]
token_meta_role_name    app200# Read KV secret
export VAULT_TOKEN=s.JGHR7HEc0I51kOvYSfR1SfGj
vault kv get kv/app200/static
====== Metadata ======
Key              Value
---              -----
created_time     2021-06-05T13:44:18.9141257Z
deletion_time    n/a
destroyed        false
version          1====== Data ======
Key         Value
---         -----
app         app200
password    cheese
username    app200

To de-board an application, simply remove the entity from the same variable and re-rerun terraform apply.

Automating the Workflow using Terraform Cloud/Enterprise

Although we used terraform CLI commands for this demo, the overall approach should be automated via a CI/CD Pipeline or GitOps Workflow. Terraform Cloud/Enterprise (TFC/TFE) supports different types of Runs which can be used for automation. Below is an example screenshot showingapp200 being added through a VCS Driven Run.

TFC UI showing app200 related resources being created

Using TFC or TFE allows us to securely maintain all terraform state file versions among other benefits.

Testing and promotion using Terraform Modules

Using an IaC approach allows us to apply configurations in non-Production, perform use-case testing, then promote to higher environments. This process can be made simpler by encapsulating Terraform vault configuration as modules, and publishing them in the in the Private Module Registry (PMR).

Testing and promoting a Terraform module for vault

In the diagram above, we are enabling the dynamic GCP credentials use-case case by expressing it in a terraform-vault-secrets-gcp module and testing it in non-production first. This approach allows supported use-cases to be developed, tested, and versioned independently — resulting in greater velocity and cleaner separation of concerns.

Conclusion

We reviewed how to create Vault configurations using a codified approach that is repeatable and can be easily automated. Using IaC as the source-of-truth also makes it easier to audit how Vault is being used. Additional best practices referenced here include using ACL Policy templates and pre-creating application entities. Please see the links below for more details on some of these best practices. Note: please run make clean , terraform destroy and docker-compose down to clean up demo configurations.