Further Terraform enhancements
Embracing a few simple changes to improve the simplicity and resilience of our Terraform solution.
Terraform is a powerful tool developed by HashiCorp to declaratively define immutable infrastructure-as-code. In my last post Bringing Terraform under control, I described how my team and I had harnessed the power of Terraform to organise our infrastructure and keep the complexity of our solution to a minimum. Since then, we’ve made a number of changes to further improve upon this.
The environments folder remains but now in a much more simplified form. Previously, there were sub-folders for each environment and each region, with a single main.tf file representing the top level of the system. This resulted in a large amount of bloat, a significant amount of duplicated code, plus the one file per folder code smell.
The reason we started with one file per environment and region was to allow us to set specific variables and configure the Terraform backend to store state.
In the new design, the death by folders environment structure is gone and each environment is represented by a single JSON file. This means that specific variables can still be set and the entry point is now a single main.tf in the project route which makes things a lot clearer.
As our infrastructure is the same in each region there is also no need for region specific variables which further reduces duplication. The region was already being passed into the go.ps1 script, to specify the Azure region when running the Terraform commands in the build pipeline, so this parameter can now also be passed into the Terraform plan as a variable to create unique resource names based on our naming conventions.
This change was made possible by shifting the responsibility of initialising the backend from the environment main.tf file to the go.ps1 script. The Terraform init command provides functionality to set up the backend using parameters which we extract from the environment JSON.
The Terraform plan is able to accept JSON as a variable file, so that same environment JSON that’s used to extract the backend parameters can be used in the plan. Having a single entry point guarantees the same blueprints are applied to every environment. These changes also allowed us to remove over 250 lines of code and make it easier to maintain, thanks to reduced duplication.
Terraform does provide a concept called workspaces which can be used to control variables for different configurations. I personally feel that workspaces can quickly become quite cluttered, especially if you need to maintain a number of environments. I also don’t think they work so well in automation so we decided to go with the approach above.
Set secrets using Terraform commands
Secrets are required to initialise the backend and authenticate with the Terraform provider. In order to prevent these secrets from being stored in the environment files, and in turn committed to source control, we were previously passing them into the go.ps1 script as parameters. The script then used PowerShell to replace the secrets in the files at runtime before reverting them back afterwards which was messy and a bit of a hack.
Now that we have harnessed the ability to run the Terraform commands with variables at runtime, there’s no longer a need to replace and set the keys using PowerShell. The secret is still passed into the go.ps1 script but it is instead passed in as a Terraform variable which makes the script less complex and more secure.
Terraform state backup
When using the Azure Terraform provider, tfstate files are stored in an Azure storage account. Whenever the state is accessed, a lock is applied to prevent more than one user from modifying the state at the same time.
The state, however, is not backed up, so users are unable to view the history of state, or, retrospectively fix state that has been modified incorrectly (this is unlikely but it’s always good practice to have a disaster-scenario plan). In order to combat this, we now manually backup the state to a separate location before running Terraform apply.
This is simply achieved by using the built-in Azure PowerShell commands to copy the state to an alternate storage account. And, as we already have the credentials set in the variable files, it makes it easy to authenticate.
Hopefully this article provided you with some ideas to help simplify your Terraform solution — however big or small the change is.
Feel free to drop me a message or give me some claps so more people can see this post.