Terraform — Securing your State file
Over the past year, I have been working on a large GCP project that involves spinning up large environments and provisioning infrastructure to fulfill developers’ needs. One of the biggest issues that we ran into was sensitive data making its way into Terraforms remote state. For example, provisioning a service account, creating a database, etc. all causes sensitive data to end up in the remote state file.
Since I was working with a team of engineers, we store Terraforms remote state file in a private GCS (Google Cloud Storage) bucket. However, our security team scans every GCS bucket within our organization looking for sensitive data, and within a few days, we had a ticket assigned to our team asking us to remove this sensitive data. Removing this data from the state file would cause some issues with our deployment; thankfully, Google allows CSEK (Customer Supplied Encryption Keys) to be provided, which enables you to encrypt your GCS objects.
What is this remote state file you speak of? For anyone new to Terraform, I would recommend reading the documentation on the purpose of a state file [1].
State file versioning
Before we configure our CSEK, I would strongly recommend enabling versioning on your GCS bucket. Enabling versioning on a GCS bucket that stores the Terraform state file is considered best practice and I would agree 100% with this; without versioning, your state file could become corrupt and a corrupt state file means no more successful Terraform runs.
Enabling versioning on your GCS bucket can be achieved by using the following gsutil command.
Once you have enabled versioning on your bucket, you can create a test file and verify the version as shown in the commands below.
As you can see, you can verify the previous version by including the version number after the hash (#) sign.
With versioning enabled, you can easily recover deleted objects. Deleted objects can be viewed using the gsutil ls -al command.
Encrypting your state
With Google’s CSEK, you must provide a key that is a 256-bit string, which is base64 encoded. This key can be generated and encoded using the following command.
When you have generated your encryption key you will want to store this key in a secure location such as HashiCorp Vault or in another secrets management system. With the encryption key, you can configure Terraform to use this key by setting an environment variable GOOGLE_ENCRYPTION_KEY or by setting the value in the backends.tf file as shown below.
Once you have configured your backend you can issue the terraform init command to initialize the backend. After you initialize your Terraform backend, you will notice in the Google console that your state file has an encryption type of customer-supplied key as shown below.
If you try to view the contents of the state file, you will get the following message: “The target object is encrypted by a customer-supplied encryption key”. This is because we did not provide our key to decrypt the content.
Viewing your state file
Since the state file is now encrypted, you cannot view the content without providing the encryption key. With the gsutil command, you can either create a .boto file with your encryption key inside or provide your encryption key on the command line. I would recommend the first option; but for testing, you can provide your key via the command line as shown below.
