Building an IaC pipeline for Terraform

For many Infrastructure as Code (IaC) is a buzz word that they hear at meet-ups, around the water cooler conversations and in meetings with vendors and consulting companies. It is often mentioned when talking about Terraform and Pulumi as vendors for implementing IaC.

In this post I’m going to show you how to setup a Terraform driven setup to deploy IaC for any service that provides a Terraform Provider.

What is IaC & Terraform?

Infrastructure as Code is, at a high level, a way to maintain hardware configuration, software configuration or any other type of configuration using a declarative mode. “Infrastructure” really means anything that can be controlled programmatically using a “provider” or an agreed on RESTful API.

This means you don’t have to click through wizards and setup screens to install, configure or change your assets, instead you write code that describe the desired configuration and with some magic behind the scenes the desired configuration is reached.

Some examples of “infrastructure” are: F5/BigIP configuration, all kinds of Azure andAWS resources as well as other software providers that choose to implement an Terraform and/or Pulumi provider, such as Okta, LaunchDarkly, Sumo Logic, New Relic to name a few that I have hands-on experience with.

At the moment the two big players in this space are Terraform and Pulumi. They have a similar philosophy but different implementations.

Terraform

With Terraform one defines the desired configuration using a tech agnostic language and Terraform uses required providers to provision resources needed to reach the desired configuration.

State

A very important aspect of managing IaC is state:

• Current state: last recorded state recorded after last deployment. This may or may not be identical with the Actual state.
• Desired state: final configuration that needs to be reached at the end of the process
• Actual state: the physical state of the “infrastructure”. This can differ from the current state if any manual change were performed outside if the IaC process. This is acceptable because IaC clients like Terraform and Pulumi know to refresh current state with actual state before comparing with desired state.

Setup

If you think about regular code deployments you need a source code (Git repo most probably) and some way to compile and deploy the code through environments (Dev, Test, Prod). The same principle applies for IaC code developed with Terraform.

Terraform Repo(Git) Structure

You have the option of using a mono-repo or individual repos for each Terraform module. I favor the mono-repo style so this is what I’m showing here.

Below is a depiction of a Terraform mono-repo with two modules, APIM and BigIP. Each module has two sub-folders: src and env .

The SRC folder contains all Terraform source code files (*.tf, *.auto.tfvars, *.auto.tfvars.json, *.json) for describing resources for that module.

The ENV folder contain individual sub-folders for each environment, such as Dev, Test or Prod which is required to test and deploy that module’s resources.

These folders contain specific values for variables and provider connectivity information for that specific environment. For example that may include Azure subscription, tenant, resource group, etc. for Dev, Test, Prod environments. Of course this information should not be hardcoded in the .tf files, instead it should be injected/substituted at runtime when the pipeline executes.

Additional contextual information for each module is highly tied to the type of resource maintained.

In a future post I will provide an example of managing IaC for Azure API Management service configuration with Terraform, which should provide a specific example.

For now here is a super simple example of this setup in which we create a local file with Terraform. The example show how to use environment variables that can be used in the Terraform code. The readme.md file contains the Terraform CLI commands to run the code.

Pipeline per Module

Using this mono-repo structure we need to isolate each pipeline to one module (apim or bigip) and depending of the capabilities of your CI/CD system you can control the environments progression through automation for planning and deploying the changes with the Terraform CLI.

Deployment Pipelines

In order to deploy any Terraform code there are at lest two steps that need to be executed: plan and apply. Additional steps can be include in between such as Policy-as-Code rules to make sure users are allowed to create/modify/delete resources or Security Scans (Snyk, tfsec, checkov, etc) to make sure no security vulneravilities are introduced in the code or other 3rd party tools.

There are few ways to run the Terraform CLI with your code:

• Run Terraform CLI from your CI/CD platform (Octopus, Teamcity, Jenkins, etc)
• Run Terraform using Terraform Cloud/Enterprise

Regardless the way you run your commands the result should be the same.

The plan command compares the code in the repo with the last recorded state after is refreshed from the actual infrastructure. Terraform then decides what changes are required to achieve the desired state.

The apply command will take the previous plan and execute it.

In between the two steps should be a hard stop for a user with proper rights to review and approve the planned changes.

Conclusion

As you can see implementing an IaC is not very complicated, it just takes some preparation and a little bit of setup.