Securing your GCP workflow using Secrets Engines on HashiCorp Vault

All too often I see the Key/Value (KV) Secrets Engine being used as a catch-all for secrets management. A perfectly viable solution, but…what if I told you there was a better way?

4 min readAug 26, 2021

Introduction

Let us assume I have a Jenkins pipeline that — as one of its many steps — provisions a VPC, some VMs, creates some firewall rules and an HTTP load balancer using Terraform. Currently, I have a static KV secret that contains the key of the service account that Terraform uses and I would rotate the keys manually once a month and update the respective KV secret…which was fine when I only had a few projects to manage, but now I find myself faced with having to rotate and update the credentials for 25+ projects. Furthermore, if someone in the IT team resigns, I now have to rotate them again!

If you are thinking: “There’s got to be a better way!”…well, there is. The answer is the GCP Secrets Engine.

I will be using Google Cloud (GCP) in my example which will have a fictitious GCP project named “My Project” and a project ID of “my-project-123”.

Prerequisite

On the GCP side, you will need to enable the APIs:

Identity and Access Management (IAM) API
Cloud Resource Manager API

And create a service account (i.e. vault-sa) in Cloud IAM with the permissions/roles of:

roles/iam.securityAdmin
roles/iam.serviceAccountAdmin
roles/iam.serviceAccountKeyAdmin

In addition, you will need to create an initial private key in JSON format for this service account.

Setup

It is time to enable the GCP Secrets Engine:

vault secrets enable -path=”my-project-123” gcp

TIP: never use the default path; use your project ID as the path for easier management and organization of your secret engines.

vault write my-project-123/config credentials=@/path/to/creds.json ttl=3600 max_ttl=21600

If you do not specify a value, then the ttl and max_ttl values will be default to the max of 768h. Here I chose to lower it to 1h and 6h respectively, meaning that all the keys this particular secrets engine produces will dissolve after an hour, with the functionality for someone with the proper permissions to extend (or renew in Vault terms) up to 5 times for a maximum of 6 hours.

NOTE: set your ttl and max_ttl values according to your own use case.

Next, I will create a roleset. A roleset is — as its name implies — a set of roles that you would like a particular service account to have. I will name my roleset “tfuser” and the type of secret that it returns is a service_account_key (the other type is access_token, but that is being deprecated):

vault write my-project-123/roleset/tfuser project="my-project-123" secret_type="service_account_key" bindings=@bindings.hcl

Where bindings.hcl is:

The resource is the scope at which the roles apply to. Here, my service account will have access at a project level. You can set the resource to be at a folder level, GCS bucket level, VM instance level, etc.

This should produce a service account in GCP with the name vault[ROLESET_NAME]-[EPOCH_TIMESTAMP] (or in our case, vaulttfuser-1629929294@my-project-123.iam.gserviceaccount.com).

How to Use

Now that you have a service account with the permissions you want, you need only to generate the keys in order to use it. Simply read from the endpoint and it will return data from which you will want the private_key_data and optionally the lease_id.

vault read -format=json my-project-123/key/tfuser

Will output something like this:

{
  "request_id": "1c55e856-d2d2-ffdf-dfdb-0669c1fef300",
  "lease_id": "my-project-123/key/tfuser/AxD0Ir2sEaCPOAJRK2Tl7BuOd",
  "lease_duration": 3600,
  "renewable": true,
  "data": {
    "key_algorithm": "KEY_ALG_RSA_2048",
    "key_type": "TYPE_GOOGLE_CREDENTIALS_FILE",
    "private_key_data": "ewogICJ0eXBlIrunInMlcnZpY2VfYWNjb3VudCIsCiAgInByb2plY3RfaWQiOikgleN....................................................................
....................................................................
....................................................................
....................................................................
....................................................................
zMSU0MHpvZXktMjE1NzEyLmlhbS5nc2VydmljZWFjY291bnQuY29tIgp9Yu=="
  },
  "warnings": null
}

The private_key_data is base64 encoded so you will have to decode to get your key. While optional, I recommend getting the lease_id and make a habit of revoking the key as a final step in your workflow rather than letting the TTL run out:

vault lease revoke [LEASE_ID]

In addition, GCP has a hard limit of 10 keys per service account so proper cleanup of unused keys will help ensure you do not hit those limits.

Next Steps

What I have outlined is a way of leveraging Secrets Engines on HashiCorp Vault to create short-lived, dynamic credentials to both secure your workflow and reduce the overhead of having to rotate those credentials. Your use case or company policies may necessitate different configurations.

To learn more about what Secrets Engines can do for you, here are some useful links:

EDIT 2021–10–26: I have added a couple of articles that builds on the scenario presented here to further enhance the security of your workflows: