Terraform
A few weeks ago I started working with Bicep, the Azure resource deployment language based on ARM, and I had really enjoyed it. Compared to the ARM templates and parameter files I have worked with previously, Bicep is much more code-based, easier to read and much easier to understand. Building IaC (infrastructure as code) was fun again.
But this was before I looked into Terraform. Compared to Terraform which is an IaC tool that works, Bicep looks to me more like it just came out of beta. Bicep is not a full replacement of the az cli and compared to Terraform it’s an inconsistent mess.
Why use Infrastructure as Code?
Infrastructure as Code (IaC) is the management of infrastructure (networks, virtual machines, load balancers, and connection topology) in a descriptive model, using the same versioning as DevOps team uses for source code. Like the principle that the same source code generates the same binary, an IaC model generates the same environment every time it is applied. IaC is a key DevOps practice and is used in conjunction with continuous delivery.
IaC is the automated way of turning the actual state (infrastructure deployed) back into the desired state (infrastructure defined in Git). It also makes it possible to prevent the actual state from drifting too far (or at all) from the desired state (environments turning into snowflakes).
What is Terraform?
Terraform is an open-source infrastructure as code software tool that provides a consistent CLI workflow to manage hundreds of cloud services. Terraform codifies cloud APIs into declarative configuration files.
Terraform, as defined above, is a IaC tool developed by Hashi Corp. Just like Bicep, Terraform is making infrastructure configuration readable and simple, by abstracting most of it and only requiring the user/developer to provide the really necessary input.
How does Terraform achieve this?
Terraform creates and manages resources on cloud platforms and other services through their application programming interfaces (APIs). Providers enable Terraform to work with virtually any platform or service with an accessible API.
While it makes it possible to create resources using their APIs on multiple cloud platforms, Terraform also keeps a state file of the latest deployment. This state file can be used to make changes comparing the current state file to the actual state, revert changes made outside of the CD process and even destroy the entire deployment.
The Terraform process provides 5 steps in total and has 3 steps at it’s core:
- Write: Create the configuration files for deployment
- Terraform Init: Initializes terraform for usage
- Terraform Plan: Creates an execution plan and validates the configuration files
- Terraform Apply: Creates the resource by applying the plan
- Terraform Destroy: Destroys the resources, can also be done on combination with plan and apply for extra insurance
Terraform and Azure
Terraform is integrated in the Azure platform. It’s possible to deploy Terraform configuration files from Azure DevOps using existing Terraform tasks. For more information check out the documentation on using Terraform for Azure. You can deploy Terraform files using az cli, powershell and Azure DevOps. If you open cloud shell in the Azure portal, terraform comes pre-installed.
There are no road-blocks to using Terraform with Azure as far as I have seen. HashiCorp even provides an entire library of configuration files for Terraform on Azure.
Why is this better than Bicep, ARM or the Azure Portal?
First of all, having your infrastructure defined in a GIT repository and deployed via a CI/CD process that is using reviewers, is the safest way to manage your production environment. When working with IaC you don’t want to touch the portal, use az cli or powershell commands (I base my experience on Azure but this is true for any cloud platform).
Bicep, as I have explained in an earlier article, is still in development when it comes to being a fully functioning IaC tool. I can give two examples to make this clear:
- When I tested it it wasn’t yet possible to add secrets to key vault from a bicep deployment. You have to use az cli, powershell or the portal.
- Bicep is inconsistent: the configuration needed for an ARM deployment can and sometimes does differ from the input needed for the same deployment using Bicep. Linux VM’s with ssh keys need also a username and password in Bicep but not when deploying with ARM.
Terraform seems to have it all sorted out. I can easily deploy resources (after the first test run has succeeded) just by providing the minimal configurations like resource group name, resource name, … etc. I can add keys to key vault during the deployment using Terraform and the language seems to be consistent with ARM, az cli and the portal. Furthermore there is the option to clean up succeeded or failed deployments using the state file stored locally or in a blob storage on Azure. It’s a fantastic tool that makes me realize I had a headache deploying anything else before I used Terraform.
Terraform, too good to be true?
Without trying to rain down on a sunny day we have to look at some of the downsides of Terraform:
- Terraform is a one way street: while deployments are done in git and are triggered by new changes to the code base, drifts cannot be detected without the help of other tools. If the infrastructure is changed outside the repository it has to be corrected by manually triggering a new deployment.
- Easy to start, hard to follow: Terraform is very readable and easy to write with the help of extensions that can be installed in your IDE. But if your resource deployment is more complicated (Airflow, VM’s in Vnets, etc) you have to do some digging. Most configurations out there (blogs, github) are for AWS and translating them to Azure requires manual work and some cloud engineering knowledge.
- There is no back to basics: A failed Bicep deployment requires a deeper look into my configuration files, some experiments in the test environment and/or the Azure portal. I can even compare my Bicep configuration with the ARM template it is based on and that I can download any time. Terraform is not that simple. Once a Terraform deployment fails and the issue is not easily found, there is nothing I can refer to. If Google fails me, I might have to spend hours figuring out how to build my infrastructure through trial and error.
Terraform, I want more!
As a data engineer working on Azure, I want more Terraform. More precisely I want to have more open-source configurations for Azure in Terraform. Especially Airflow, why is the good stuff restricted to AWS?
Terraform is a fantastic tool, that solves a lot of DevOps problems. It’s multi-cloud, easy to read and write and user-friendly. The possibility to clean up my resources blew me away the first time I used it.
The one question I am left with is the issue with drifts. How do we manage manual changes in infrastructure and how do we revert them back to the desired state when working with Terraform?
I hope you enjoyed this introduction to Terraform and also learned something new. If I missed out on anything important, made mistakes or got it wrong please let me know in the comments. As always all questions, suggestions or comments are welcome.
If you found this article useful, please follow me.
