How to run Terraform like a pro, in a DevOps world.

If you have 10+ environments, including multiple tenants and/or your team follows the sun, continue reading.. otherwise, read it anyway, this is how everyone should work with Terraform and avoid having tfstate files corruption.

Having worked with Terraform for the past 7 months I can confirm, is one of the best tools out there, with few drawbacks when we talk about tens of runs every day per environment (Yes, we have 10+ environments, from staging to production, more than 4 tenants) and our code iterates about 10 to 15 times every day.

Let’s rewind a little bit on how Terraform works.

Terraform code is written in a language called HCL in files with the extension “.tf”. It is a declarative language, so your goal is to describe the infrastructure you want, and Terraform will figure out how to create it.

Terraform Pieces:

• TF files: This is the “code” you create to spin up the infrastructure.

resource "aws_instance" "mywebserver" {
  ami = "ami-12ab56cd"
  instance_type = "t2.micro"
}

• TF vars: These are the variables you can use to reference different environments when you run your terraform plan or apply and you call it like using -var-file=vars.tfvars
• TF state files: Terraform keeps track of all the resources it already created for this set of templates, so it knows your EC2 Instance already exists when you run it again. All this is kept in state files. When you run terraform plan it gets the latest state, it compares with what you want to do and gives you a result of what will happen if you run terraform apply .

The main problem resides when you have multiple people running plan and apply at the same time, so you need remote state files to be downloaded and compared with.

How do we fix this problem?:

Luckily for us, there are other tools out there to help us with this and not being very hard to implement it. One of these tools is Terragrunt. With Terragrunt someone can lock an environment state whilst planning or applying changes, and I will show you how we implemented it.

This work has been achieved by my team: Ebrahim Moshaya, Joshua Wright, Chris Funderburg, Melvyn Mathews and Leigh Hayward so full recognition to them!

We use a Makefile to run all our plans and applies, I had to modify some values to preserve the confidentiality.

define ACCOUNT_IDS
"\033[1;31m-------------------------------------\033[0m"
@echo -e "\033[1;32m123456789012 - Customer 1\033[0m"
@echo -e "\033[1;32m123456789012 - Customer 2\033[0m"
@echo -e "\033[1;32m123456789012 - Customer 3\033[0m"
@echo -e "\033[1;31m-------------------------------------\033[0m"
endef
################################################
# VARIABLES
################################################
PARALLELISM = 10
TENANT = tenant
ENV = env
################################################
# TENANTS
################################################
customer1:
 $(eval TENANT = customer1)
customer2:
 $(eval TENANT = customer2)
customer3:
 $(eval TENANT = customer3)
################################################
# ENVIRONMENTS
################################################
dev:
 $(eval ENV = dev)
dev2:
 $(eval ENV = dev2)
test:
 $(eval ENV = test)
test2:
 $(eval ENV = test2)
uat:
 $(eval ENV = uat)
pre:
 $(eval ENV = pre)
prod:
 $(eval ENV = prod)
qa:
 $(eval ENV = qa)
################################################
# TARGETS
################################################
account-ids:
 @echo -e ${ACCOUNT_IDS}
update-modules:
 rm -rf .terraform
 terraform get
plan:
 export TENANT=${TENANT} && export ENV=${ENV} && export AWS_PROFILE=default && terragrunt plan -var-file="tfvars/$(TENANT).$(ENV).tfvars" -module-depth=-1 -parallelism=${PARALLELISM} ${ARGS}
apply:
 export TENANT=${TENANT} && export ENV=${ENV} && export AWS_PROFILE=default && terragrunt apply -var-file="tfvars/$(TENANT).$(ENV).tfvars" -parallelism=${PARALLELISM} ${ARGS}
################################################
# TERRAFORM ACTIONS
################################################
## CUSTOMER1-DEV ##
customer1-dev-plan: customer1 dev update-modules plan
customer1-dev-apply: customer1 dev update-modules apply
## CUSTOMER1-DEV2 ##
customer1-dev2-plan: customer1 dev2 update-modules plan
customer1-dev2-apply: customer1 dev2 update-modules apply
## CUSTOMER2-DEV ##
customer2-dev-plan: customer2 dev update-modules plan
customer2-dev-apply: customer2 dev update-modules apply
## CUSTOMER3-DEV ##
customer3-dev-plan: customer3 dev update-modules plan
customer3-dev-apply: customer3 dev update-modules apply

How to use it?, just run commands like:

• make customer1-dev-plan
• make customer1-dev-apply

This is nice, but it won’t work until we install terragrunt. After you install terragrunt, and configure it as explained here the only thing left is create a proper .terragrunt file within your repository.

# Configure Terragrunt to use DynamoDB for locking
lock = {
  backend = "dynamodb"
  config {
    state_file_id = "devops-terraform.${get_env("TENANT", "customer1")}.${get_env("ENV", "dev")}"
    aws_region = "eu-west-1"
    table_name = "terragrunt_locks"
    max_lock_retries = 360
  }
}
# Configure Terragrunt to automatically store tfstate files in an S3 bucket
remote_state = {
  backend = "s3"
  config {
    encrypt = "true"
    bucket = "${get_env("BUCKET_PREFIX", "cloud")}-${get_env("TENANT", "customer1")}-${get_env("ENV", "dev")}-tfstates"
    key = "devops-terraform/${get_env("TENANT", "customer1")}.${get_env("ENV", "dev")}.tfstate"
    region = "eu-west-1"
  }
}

If you pay attention carefully to the piece of code above, you will notice we are referencing environment variables that are coming from the Makefile, and this allow us to “switch” tenants and customers really easy.

I still remember the days when I joined the team and we were announcing things like this in Slack: “hey I have the customer1-dev tfstate!!!!, don’t run terraform there!!!”

Using Terraform and Terragrunt like this, you can also integrate this to a Jenkins Pipeline or AWS CodePipeline, but that is conversation for another post.

I hope you can implement this and let us know if it worked as smoothly as it worked for us, as this is saving us a lot of headaches.

— Fernando