A Moving Window of Trust

How to securely build and deploy with dynamic credentials

Published in

HashiCorp Solutions Engineering Blog

6 min readApr 26, 2021

Deployment pipelines are a focus for attackers and a source of stress for DevOps engineers when managing credentials. It seems that there is just no good answer out there.

We spend our time wondering how we can make it better and more secure, knowing that there is always one or more static credentials spread across a few secure/encrypted places.

HashiCorp Vault is a prominent choice for centrally managing secrets in your apps and infrastructure, but if you’re just starting out, you may not be taking advantage of dynamic secrets. While Vault is great for static credentials too, the ultimate goal of using Vault is to take those hardcoded credentials and turn them dynamic — meaning we want to automatically rotate them.

By making all credentials dynamic, we create a window of trust in which an application is built, deployed, and eventually runs. This window moves from the application build/deploy stage to the infrastructure it runs on, pulling its own dynamic secrets from Vault.

Have a quick look at this Vault access pattern.

The power of dynamic secrets is that if a credential leaks, it’s only useful during the time when that window of trust is open — which you can eventually make shorter and shorter.

In this article, I’ll show you how to use Vault dynamic secrets along with GitHub Actions and Terraform Cloud to build a secure deployment pipeline where even leaked credentials are useless after a short period of time.

The Setup

In step 2 of the “Application using Vault Auth backend” diagram, the process uses a 3rd party for identity confirmation. In this tutorial, I’ll show how you can do this with GitHub Actions or Terraform Cloud.

GitHub Actions

Application: GitHub runner
Application Identity: Runner token; Run ID; Run number; owner; repository
3rd Party/Orchestrator: GitHub

Terraform Cloud / Enterprise

Application: Terraform worker
Application Identity: Worker TFE token; Run ID; Workspace name
3rd Party/Orchestrator: Terraform Cloud / Enterprise

For both of these backends, the parameters used to define an identity contain dynamic fields and only allow access to active “runs”, which means that any kind of leak would be inconsequential since those details would no longer work! The window of trust is only open for that single run.

Why Do We Need GitHub Actions?

With VCS backed workspaces, TFC and TFE can trigger Terraform Plan and Apply actions on a PR merge.

However, for API/CLI backed workspaces, something needs to trigger these Terraform actions and some developers have their application code build and deployment pipeline in the same repo, so some prefer to have this level of control.

For VCS backed TFC/TFE workspaces, you can just skip the GitHub Actions-related sections.

Embrace the Plugin

Vault authentication backend is highly pluggable and allows you to extend it to suit your needs. Writing a Vault plugin is not that difficult and HashiCorp has a great tutorial on this — https://learn.hashicorp.com/tutorials/vault/plugin-backends

To achieve our objectives we’ll be using HashiCorp Vault as our centralized secrets backend with 2 plugins:

GitHub actions authentication plugin: https://github.com/Mongey/vault-plugin-auth-github-actions
TFE authentication plugin: https://github.com/gitrgoliveira/vault-plugin-auth-tfe

For the rest of our setup, as you guessed, our infrastructure code is hosted on GitHub and we’ll be using Terraform Cloud for AWS deployments.

Here’s an overview of the full end to end workflow

Implementation Steps

If you made it this far and would like to implement this workflow on your pipeline, follow the next steps.

Step 0: Download the respective plugins

Please make sure you pick the latest version and the appropriate architecture.

https://github.com/Mongey/vault-plugin-auth-github-actions/ — you will need to build this plugin from source.
https://github.com/gitrgoliveira/vault-plugin-auth-tfe/releases/tag/0.0.10

The binaries should be added to a plugins folder, where Vault will load them from.

Step 1: Start Vault

You can start Vault locally in dev mode. Please be sure that you are targeting the correct plugin folder and exposing the Vault listener on an IP address that is reachable by both GitHub and TFC.

vault server -dev -dev-root-token-id=root \
    -dev-plugin-dir=./plugins -log-level=debug \
    -dev-listen-address=x.x.x.x:8200

Plugins will be automatically loaded in dev mode, but to run them in a production cluster, follow these instructions — https://www.vaultproject.io/docs/internals/plugins#plugin-catalog

Step 2: Enable Terraform Cloud Secrets Backend

This backend is available with Vault 1.7.

HashiCorp has a tutorial that takes you step-by-step on how to set this up— https://learn.hashicorp.com/tutorials/vault/terraform-secrets-engine

Step 3: Setup GitHub Actions Plugin

vault policy write github-policy - <<EOF
path "secret/data/*" {
  capabilities = ["read","create", "delete", "update"]
}
path "secret/*" {
  capabilities = ["read", "create", "delete", "update"]
}
EOFvault auth enable \
    -path="github-actions" \
    vault-plugin-auth-github-actionsvault write auth/github-actions/repositories/<owner>/<repo> \ 
    policies=github-policy

This configuration is establishing a relationship between the GitHub repo and which secrets can be used.

Step 4: Test GitHub Actions Plugin

Example of a GitHub Action using Vault authentication.

name: ImageBuilder
on: push
jobs:
 build:
   runs-on: self-hosted
   steps:
     - uses: actions/checkout@v2
     - name: Import Secrets
       id: secrets
       uses: hashicorp/vault-action@v2.1.2
       with:
         url: http://127.0.0.1:8200
         tlsSkipVerify: true
         method: github-actions
         authPayload: |
           {
             "token": "${{ secrets.GITHUB_TOKEN }}",
             "run_id": "${{ github.run_id }}",
             "run_number": "${{ github.run_number }}",
             "owner": "${{ github.repository_owner }}",
             "repository": "${{ github.repository }}"
           }
         secrets: |
           terraform/creds/actions token | TFC_TEAM_TOKEN    - name: Setup Terraform
      uses: hashicorp/setup-terraform@v1
      with:
        cli_config_credentials_token: ${{ steps.secrets.outputs.TFC_TEAM_TOKEN }}
        terraform_wrapper: false    - name: Terraform Init
      run: terraform init -input=false
      env:
        TF_WORKSPACE: "dev"
        TF_IN_AUTOMATION: "true"    - name: Terraform Format
      run: terraform fmt    - name: Terraform Plan
      run: terraform plan -input=false
      env:
        TF_WORKSPACE: "dev"
        TF_IN_AUTOMATION: "true"    - name: Terraform Apply
      run: terraform apply -input=false -auto-approve
      env:
        TF_WORKSPACE: "dev"
        TF_IN_AUTOMATION: "true"

In this example, the TFC token is being read from Vault and added as a GitHub environment variable. More examples are available in the Vault Actions repo — https://github.com/hashicorp/vault-action#example-usage

You can then use this token with the Terraform CLI or directly with the API.

HashiCorp also has a tutorial focused on Terraform GitHub Action, using Terraform Cloud—https://learn.hashicorp.com/tutorials/terraform/github-actions

Step 5: Setup TFE Auth Plugin

Create a Vault policy

vault policy write terraform-policy - << EOF
path "auth/token/create" {
   capabilities = ["update"]
}
 
path "secret/data/*" {
 capabilities = ["read","create", "delete", "update"]
}
path "secret/*" {
   capabilities = ["read", "create", "delete", "update"]
}
 
path "aws/sts/deploy" {
 capabilities = ["read"]
}
 
EOF

Setup Vault auth backend

vault auth enable -path=tfe-auth vault-plugin-auth-tfe
vault write auth/tfe-auth/config organization=<your TFC org>
 
vault read auth/tfe-auth/config
vault write auth/tfe-auth/role/workspace_role workspaces=* \
   policies=default,terraform-policy
 
vault read auth/tfe-auth/role/workspace_role

The auth backend works with TFC by default. To add your own TFE host, append terraform_host=https://<tfe hostname> to the auth backend configuration.

workspaces can be a glob * or a list of workspaces allowed to use that role.

Step 6: Test TFE Auth Plugin

This example Terraform code retrieves AWS credentials but can be adapted to any Vault secrets backend.

variable "TFC_WORKSPACE_NAME" {
  type        = string
}
variable "TFE_RUN_ID" {
  type        = string
}provider "vault" {
  address    = "http://vault_address:8200"
  token_name = "terraform-${var.TFE_RUN_ID}"
  auth_login {
    path = "auth/tfe-auth/login"
    parameters = {
      role      = "workspace_role"
      workspace = var.TFC_WORKSPACE_NAME
      run-id    = var.TFE_RUN_ID
      tfe-credentials-file = filebase64("/tmp/cli.tfrc")
    }
  }
}data "vault_aws_access_credentials" "creds" {
  backend = "aws"
  role    = "deploy"
  type    = "sts"
}provider "aws" {
  region     = var.region
  access_key = data.vault_aws_access_credentials.creds.access_key
  secret_key = data.vault_aws_access_credentials.creds.secret_key
  token      = data.vault_aws_access_credentials.creds.security_token
}

TFC_WORKSPACE_NAME and TFE_RUN_ID are set by TFC/TFE via environment variables so you only need to declare them as input to receive these values.

tfe-credential-file has a fixed path within TFE/TFC. If you are using external agents in that workspace, then use this quick snippet instead

locals {
  file_path_apply = "/root/.tfc-agent/component/terraform/runs/${var.TFE_RUN_ID}.apply/cli.tfrc"
  file_path_plan  = "/root/.tfc-agent/component/terraform/runs/${var.TFE_RUN_ID}.plan/cli.tfrc"
  file_path       = fileexists(local.file_path_apply) ? local.file_path_apply : local.file_path_plan
}

Step 7: Magic!

Your GitHub Action YAML will now call upon TFC to run the Terraform code from step 6 in your repo, which will reach out to Vault to fetch temporary credentials.

Hope you enjoyed this blog post! This workflow can be made a bit simpler, but it all depends on your requirements.

Many thanks to Russ Parsloe for his contribution to this blog post.