Provisioning Vault encrypted secrets using Terraform (using sumup-oss/vaulted and sumup-oss/terraform-provider-vaulted) 🎉

What the ordinary HashiCorp Terraform flow looks like

https://github.com/sumup-oss/terraform-provider-vaulted#typical-terraform-workflow

Using sumup-oss/vaulted, we can have the following HashiCorp Terraform flow

https://github.com/sumup-oss/terraform-provider-vaulted#terraform-workflow-with-usage-of-httpsgithubcomsumupvaulted-and-this-provider

Intro

It’s my pleasure to present you a workflow that I worked on for a while now.

At my employer, SumUp we use HashiCorp Terraform to provision infrastructure, Ansible for ad-hoc provision and HashiCorp Vault to store our secrets. We use a lot of other software too, but it’s irrelevant to this article.

It’s a PCI-DSS-compliant company and aims to do the best in terms of security for infrastructure and processes.

The tooling over the span of more than a year was enhanced with projects that we use in every possible environment that utilizes HashiCorp Terraform and HashiCorp Vault.

How we did it before: we were provisioning HashiCorp Vault using AWS Lambdas, plaintext-values, Ansible, Terraform (resource.vault_generic_secret) and other workarounds.

Problems:

  • Fragmentation of technologies and no-clear technology that does it best.
  • Multiple ways to provision HashiCorp Vault with different security issues.
  • Requires you to use different technologies that you don’t always need. E.g “why do we need an AWS Lambda if we’re not deploying in AWS (on-premise is also used in certain environments)?”, “why do I need Ansible when I only want to provision infrastructure and Consul + Vault?“, etc.

Solution points (what I focused on):

  • Find a technology/tool that we always use when provisioning environment.
  • Store everything as Infrastructure as Code while applying security common sense.
  • Find a workflow that enables non-authorized people to provide encrypted secrets that can be reviewed and applied by authorized people.
  • Find a workflow that enables only authorized people (system administrators, DevOps (enabled) engineers, site-reliability engineers, CI systems to decrypt them.

Solution:

  • I stuck with HashiCorp Terraform which we always use to provision infrastructure, also use for Consul provisioning and is easily extensible via providers.
  • Again, HashiCorp Terraform enables us to have everything as Infrastructure as Code.
  • I went forward with a combination of cryptography algorithms RSA and AES 256 that enables both asymmetric encryption and large file encryption/decryption.

Note, further in the article I tackle in details technology choice problems and their solutions.



Source Code:

https://github.com/syndbg/terraform-provider-vaulted-example

First, why HashiCorp Terraform?

The terraform documentation is a sufficient and exhaustive resource why. Head to https://www.terraform.io/guides/index.html.


Why write a cryptography tool (CLI) and custom Terraform provider?

Vaulted, the CLI. (https://github.com/sumup-oss/vaulted#why):

  • Ease-of-use.
  • First-class terraform support.
  • Asymmetric encryption. (Private key-less encryption, using only a public key.)
  • Large files are supported due to AES256 GCM encryption/decryption used.
  • GPG/PGP keychain-less which means you don’t need external GPG/PGP keychain and neither do your users. (Support for this may be added in the future)
  • Completely testable and high test coverage consisting of unit, integration and e2e tests.
  • Encryption,
  • Decryption,
  • Secret rotation,
  • Secret re-keying.

Terraform-provider-Vaulted (ref: https://github.com/sumup-oss/terraform-provider-vaulted/blob/master/USE_CASES.md#use-cases):

  • providing first-class Vault encrypted resources,
  • extremely secure measures to never log, store plaintext values in terraform state, stdout or stderr. (If there are such exceptions, email/message maintainer),
  • ease-to-use Vault encrypted resources in plain-simple terraform.
  • provides necessary security tools to rotate or rekey secrets.

Since I mentioned earlier that being PCI-DSS-compliant is a must and doing the best in terms of security also, just having a Terraform provider that communicates with Vault (such as the official one) is not enough for our use-cases.

We want to have as much encryption as possible and even avoiding terraform state having plaintext values.



Let’s use vaulted and terraform-provider-vaulted in practice

We’re going to apply the steps explained https://github.com/sumup-oss/terraform-provider-vaulted/blob/master/USAGE.md#simple-integration-approach.

Again, source code is available at https://github.com/syndbg/terraform-provider-vaulted-example.

Installation: Download and setup the terraform provider vaulted. Unfortunately, since it’s not an official Terraform provider (yet?), it has to be downloaded and installed locally manually. Check out https://github.com/sumup-oss/terraform-provider-vaulted#installation for setup instructions.

First, let's generate an RSA key pair (using OpenSSL) that we’re going to use for this example. In real company usage, the generated key pair’s private key will go into an internal company HashiCorp Vault, LastPass, 1Password, etc password manager). The public key would remain public and be committed in the git repository so that anyone can contribute secrets.

# Generating the RSA keypair and also using PKCS#1 ! Other formats are not supported yet
# Generate PKCS#1 private key
> openssl genrsa -f4 -out private.pem 4096
# Generate from private key, a public key
> openssl rsa -in private.pem -outform PEM -pubout -out public.pem

Second, let’s run the official Vault 1.1.0 Docker image. This is the Vault that we’re going to provision with secrets.

# Start a Vault server *in the background*.
# It'll be exposed at 8200 and have Vault root token of `myroot`.
> docker run -d -p 8200:8200 \
--cap-add=IPC_LOCK \
--name=vaulted_vault_example \
-e 'VAULT_DEV_ROOT_TOKEN_ID=myroot' \
-e 'VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200' \
vault:1.1.0

Third, let’s setup the provider to communicate with Vault. Create a file main.tf and write the following contents (ref: https://github.com/sumup-oss/terraform-provider-vaulted/blob/master/USAGE.md#provider):

# main.tf
provider "vaulted" {
# NOTE: Address of the Vault we just ran at second step
address = "http://localhost:8200"
# NOTE: Token of the Vault we just ran at second step
token = "myroot"
# NOTE: Private key that we generated at first step. This is going to be decrypt the encrypted Vault secrets during terraform apply.
private_key_path = "./private.pem"
}

Fourth, run terraform init expected output is:

Initializing provider plugins...
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

If you have issues finding the provider vaulted , make sure that you’ve previously installed it as explained above before the first step.

Fifth, add a plain text secret that we’re later going to encrypt. Since Vault is using JSON API, it’s expected that our secret is also in JSON that’s going to be saved in Vault as it is. I’ll create a file secret.txt with contents:

{
"example": "vaulted"
}

Sixth, create a Terraform provider Vaulted resource for Vault encrypted secret via vaulted CLI. You can do it with vaulted terraform new-resource in a single command. The alternative is to use vaulted encrypt and manually create the terraform resource, copy the encrypted payload there, etc. The terraform commands are saving us time.

# We use the `public.pem` key that we created at step 1.
# We create a Vault secret at path `secret/example`.
# We name the terraform resource `vaulted_test`.
# We use `secret.txt` as input that we're encrypting that we created at step 5.
# The Terraform HCL using the encrypted Vault secret will be created at `vaulted.tf`. Without specifying `--out` it'll be printed to stdout and you can create your own file or use another existing one.
> vaulted terraform new-resource --public-key-path=./public.pem --path=secret/example --resource-name=vaulted_test --in=secret.txt --out=vaulted.tf

Seventh, Run terraform plan. A plan must be generated successfully.

> terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
+ vaulted_vault_secret.vaulted_tf_vaulted_test
id: <computed>
path: "secret/example"
payload_json: <sensitive>
Plan: 1 to add, 0 to change, 0 to destroy.
------------------------------------------------------------------------
Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

Eight, terraform apply 🎉.

# Automatically approve without user prompt
> terraform apply -auto-approve
vaulted_vault_secret.vaulted_tf_vaulted_test: Creating...
path: "" => "secret/example"
payload_json: "<sensitive>" => "<sensitive>"
vaulted_vault_secret.vaulted_tf_vaulted_test: Creation complete after 0s (ID: secret/example)
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Nine, did it work? Yes!

# Exec inside the Vault container that we created at step 2.
> docker exec -ti vaulted_vault_example sh
/ # export VAULT_TOKEN=myroot
/ # export VAULT_ADDR=http://localhost:8200
/ # vault read secret/example
WARNING! The following warnings were returned from Vault:
* Invalid path for a versioned K/V secrets engine. See the API docs for the
appropriate API endpoints to use. If using the Vault CLI, use 'vault kv get'
for this operation.
/ # vault kv get secret/example
====== Metadata ======
Key Value
--- -----
created_time 2019-04-13T01:04:27.690880911Z
deletion_time n/a
destroyed false
version 1
===== Data =====
Key Value
--- -----
example vaulted

Conclusion

This must be sufficient to get you started with provisioning Vault secrets securely using Terraform.

Links that are going to be helpful for you and your company to cover some security considerations and questions that may pop-up:

Something wrong and not working? Feel free to comment here and open issues wherever applicable. Useful links: