Stop being selfish ! — Open up Terraform to your team with Atlantis

Florian Dambrine
Jun 25 · 8 min read

What is Terraform ?

is a powerful software from that allows you to write in a declarative way to provision and manage any cloud, infrastructure, or service.

Image for post
Image for post
Terraform Workflow (Source: )

The terraform plan command (similar to git diff output) is used to create an execution plan. Terraform performs a refresh, and then determines what actions are necessary to achieve the desired state specified in the source code.

Image for post
Image for post
Output generate by terraform plan command (Source: )

Once the execution plan looks good, terraform apply command is used to apply the changes required to reach the desired state of the configuration according to the plan.

Ops engineers are usually responsible of operating such infrastructure software, but what if we could change that and unblock developers to build even faster ?

Infrastructure ownership ?

It is well known that DevOps engineers are usually known as “Gods” due to their high set of privileges to operate a cloud environment. Tools like Terraform often requires this set of privileges specially if the goal is to capture 100% of resources as code (IAM, EC2, RDS, R53, …).

As powerful as it can be, Terraform can also be heavily destructive if not used wisely. This is why most of the time its usage is restricted to a limited amount of engineers (mostly DevOps) within the company.

Part of the daily routine of a DevOps / Automation engineer working with Terraform is to write a (a set of resources logically grouped together) that captures the needs of a new piece of software in order to automate its setup.

Modules are what libraries are to software engineers — A set of functions that can be called with input parameters.

Let’s take an example to better illustrate this statement. Here is an architecture diagram of an existing AWS environment (green) and the infrastructure required by a new project (yellow) that needs to be captured as code. Let’s assume that the green environment is an image metadata API and the yellow one is a new feature to process images metadata asynchronously and store them in S3:

Image for post
Image for post
Infrastructure diagram of an existing environment on the left and a new project requirements on the right

The resulting Terraform module may look like this:

terraform-modules
├── autoscaling # Module used to setup green infra
│ └── *.tf
├── database # Module used to setup blue infra
│ └── *.tf
├── load-balancer # Module used to setup purple infra
│ └── *.tf
---------------------------------------------------------------
# New module
--------------------------------------------------------------
└── new-project # New module to setup yellow infra
├── s3.tf. #-- Create S3 bucket
├── sqs.tf #-- Create SQS queue
├── outputs.tf. #-- Return important values to the caller
└── variables.tf #-- Input variables provided by module users

A module can be instantiated that way using (a thin wrapper for Terraform):

# File: mycloud/production/async-images/terragrunt.hcl# Terragrunt will copy the Terraform configurations specified by the
# source parameter.
terraform {
source = "<path>/terraform-modules//new-project")}"
}
# These are the variables (defined in variables.tf) we have to pass
# in to use the module specified in the source above.
inputs = {
environment = "production"
s3_bucket_name = "s3-async-images"
sqs_queue_name = "sqs-async-images"
}

Wait a minute… The skills required to write a Terraform module is surely on the Ops team, but any developer with minimum guidance and tooling can easily write the above file (<10 lines worth of content) or contribute to such a repository !

If tomorrow your team needs a similar setup for asynchronous page and video processing. It is as trivial as making new calls to the same module but with different input variables !


There are a lot of advantages opening up repositories to your developer teams:

  • Raise awareness on the infrastructure setup helping developer to better understand how all pieces stick together.
  • Reduce time to troubleshoot issues as developers gained a global vision on the overall infrastructure.
  • Remove potential blockers by enabling more people to do Ops work.
  • Constraint Ops to think about real code practices for Terraform (Pull request, code reviews, CI / CD pipeline, no Terraform from laptop).

Let’s see how you can enable anyone in your team run Terraform/Terragrunt automations safely !

Here comes Atlantis !

Image for post
Image for post

is an application for automating Terraform via pull requests. It is deployed as a standalone application into your infrastructure. No third-party has access to your credentials.

It listens for , or webhooks about Terraform pull requests. It then runs terraform plan and comments with the output back on the pull request.

When you want to apply, comment atlantis apply on the pull request and Atlantis will run terraform apply and comment back with the output.

Atlantis Pull request review workflow on Bitbucket
Atlantis Pull request review workflow on Bitbucket
Atlantis Pull request workflow on Bitbucket Cloud

I won’t detail much how to get Atlantis up and running as it’s already well explained in the . Here is a high level overview of how Atlantis interact in your workflow:

Image for post
Image for post
Atlantis Workflow

At GumGum the Machine Learning Engineering team operates an infrastructure made of 80+ Terragrunt instantiation (provisioning ~1500 ) using 20 Terraform modules across multiple regions and accounts.

From a simple Cloudwatch Alarm to the , Terraform manages our infrastructure on a daily basis.

Given that the volume of resources managed as code and as a former Senior DevOps Engineer who transitioned back to Software engineering (at 30%), it was more than obvious to enable the team to do Terraform on their own using Atlantis.

Because the purpose of this post is “stop being selfish !”, I will share some interesting tweaks we did to run Atlantis at scale.

Custom Workflows with Atlantis

As mentioned earlier we use to keep our Terraform code DRY and maintainable but can Atlantis fit our needs ?

  • Yes ! Atlantis developers thought about this use case and offered a way to define ! It’s a matter of adding the Terragrunt binary to the official atlantis docker image

Here is the custom we use in production that you can find on :

### Dockerfile# Grab image to collect tools from
FROM lowess/terragrunt:0.12.24 as tools
# Official atlantis image
FROM runatlantis/atlantis:v0.13.0
COPY --from=tools /usr/local/bin /usr/local/bin
COPY --from=tools /opt/.terraform.d /opt/.terraform.d
COPY --from=tools /root/.terraformrc /home/atlantis/.terraformrc
COPY --from=tools /root/.terraformrc /root/.terraformrc

Rendering atlantis.yaml on the fly

Atlantis relies on an atlantis.yaml configuration file, located under your repository root folder (this is what defines which repositories should be under its governance).

If you want the ability to run Terraform modules individually you will need to list every single path where you have module instantiations...

A reason to do that is to speed up Atlantis plan / applyas it will just act on an individual module instead of trying to plan-all / apply-all an entire folder of modules (Does not hurt but runs for modules that were not changed)

### <repo> terragrunt layout<repo>/<path>/prod/cloudwatch-alerts/
├── dynamodb-pages-reads
│ └── terragrunt.hcl
└── dynamodb-images-reads
└── terragrunt.hcl

### <repo>/atlantis.yaml---
version: 3
automerge: trueprojects: - name: dynamodb-pages-reads
dir: ./<path>/prod/cloudwatch-alerts/dynamodb-pages-reads
workflow: terragrunt
autoplan:
enabled: true
when_modified:
- "./terraform/modules/**/*.tf"
- "**/*.tf"
- "**/terragrunt.hcl"
- name: dynamodb-images-reads
dir: ./<path>/prod/cloudwatch-alerts/dynamodb-images-reads
workflow: terragrunt
autoplan:
enabled: true
when_modified:
- "./terraform/modules/**/*.tf"
- "**/*.tf"
- "**/terragrunt.hcl"
...
[A LONG LIST OF PROJECTS]
...

Because it’s just inhuman to maintain such a file manually (ours is ~400 lines), here is a that you can use to generate your atlantis.yaml file with or a simple shell script:

Gist for atlantis.yaml auto-generation

Speed-up Atlantis plan / apply

With there was the introduction of parallelism options (parallel_plan / parallel_apply ) 🎉 Yeah !

Unfortunately this feature is for now limited to which are not used by Terragrunt… As per this it seems doable “in theory” to do it with Terragrunt but unfortunately I did not get luck and ran into unreliable Terraform init steps…


Feel like we can’t do better ? Well there are other knobs to tweak !

Thanks to this great article from (Which I recommend you to read) We decided to pre-bake all the plugins we are mostly using in terraform so that they don’t need to be downloaded from the .

Given that it is a recommended practice to constrain in modules, we can easily pre-download the ones we use all over the place in order to make them directly available to Terraform. You can find an example in this .

This literally speeded up Atlantis feedback loop on pull requests !

Conclusion

Running Terraform from CI / CD pipelines is far from being trivial if you try to do it on your own… Fortunately Atlantis came to the rescue to ease this process.

As of today I have seen great engagement and enthusiasm from my peers discovering DevOps daily basis operations !

If you want to be successful with this product here is my two cents:

  • Run a POC on Atlantis ()
  • Identify low risks infrastructure pieces that can be controlled by Atlantis workflow
  • Onboard your developers by doing a high level knowledge transfer on Terraform / Terragrunt
  • Make sure to of an Ops person before applying code in production
  • Enjoy a better experience and a full developer worrkflow moving forward !

gumgum-tech

Thoughts from the GumGum tech team

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store