A guide to Terraform refactoring: Step 1 — Split your files

Hi everyone, in this series of articles we will see how to refactor Terraform code, from a basic file handling all the resources to a modular architecture. All the code used in the article can be found here : https://github.com/esker-software/terraform-refactoring.
The code uses the Azure provider because that’s what I work with everyday but the refactoring techniques described in the article are not specific to Azure.

We will start from some basic Terraform code where all your resources have been defined in a single file. The code can be found in the 1-single_file folder, it describes a simple architecture consisting of :

• a resource group (high level resource container in Azure)
• a virtual network with a single subnet (Do not try this at work! This is not a proper way to handle your network but it will be enough for us in this article)
• a “web” virtual machine that will symbolize a web server with its network interface, availability set and a data disk attached to it
• a “db” virtual machine that will symbolize a database server with its network interface, availability set and a data disk attached to it

We can now create all this infrastructure by running Terraform on our favourite Azure subscription.
Note: You might need to run terraform apply twice to create everything because the Azure provider doesn’t handle the waiting time for the resource group creation properly, leading to errors like:

Error: Error Creating/Updating Virtual Network “my-vnet” (Resource Group “test_rg”): network.VirtualNetworksClient#CreateOrUpdate: Failure sending request: StatusCode=404 — Original Error: Code=”ResourceGroupNotFound” Message=”Resource group ‘test_rg’ could not be found.”

We’re now happy, our infrastructure has been created and is managed by code, all is perfect and they coded happily ever after.
Well… not quite !
You probably already know that keeping a lot of code in one file brings quite a few problems. The readability of the code quickly diminishes with the number of lines added. Change management also becomes an issue if multiple people start working on the same file, conflicts can rapidly happen.
In addition to that, it’s much easier to handle your Terraform code when keeping multiple state files delimited to a functional perimeter. You don’t want the state of a virtual machine impacting you when you’re trying to make changes on the network.
To start fixing those issues, we will split the code between different folders each managing a functional scope:

• the resource group
• the network
• the web server
• the database server

You can find this split in 2_split_folders.
Now that your code is split properly, comes the issue of managing the Terraform state. So far you had only one file handling the state of all your resources but we will now need one state by folder. There is mainly two ways to do this:

• start from an empty state and import the resources in it using Terraform import commands
• copy the old state file and remove the resources we don’t want to manage anymore using Terraform state commands

Both solutions can work but depending on the situation they can be more or less difficult to execute.
Let’s see how we implement them on our example.

Regenerating the state: using import commands

For a simple example, regenerating the state for the resource_group folder is easier using an import.
We only define one resource in this new folder, running a terraform init and plan tells us this:

Terraform will perform the following actions:

# azurerm_resource_group.resourcegroup will be created
+ resource “azurerm_resource_group” “resourcegroup” {
+ id = (known after apply)
+ location = “westeurope”
+ name = “my_rg”
}

Plan: 1 to add, 0 to change, 0 to destroy.

Since our resource group already exists we just have to run the corresponding import command:

terraform import azurerm_resource_group.resourcegroup /subscriptions/<your-subscription-id>/resourceGroups/test_rg

Note: The syntax of the import command for most resources can be found at the bottom of the resource documentation, for example for the resource group: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/resource_group#import
Running a new terraform plan now tells us no changes are needed!

This was an easy example but the idea stays the same when working on a folder defining more resources.
Let’s take a look at the web folder containing the resources for our web server.
If we run a terraform plan, here is the list of resources Terraform wants to create:

# azurerm_availability_set.web-avset will be created
# azurerm_linux_virtual_machine.web-vm will be created
# azurerm_managed_disk.web-datadisk will be created
# azurerm_network_interface.web-vm-nic will be created
# azurerm_virtual_machine_data_disk_attachment.web-diskattachment will be created

Based on this we can craft the necessary terraform import commands:

terraform import azurerm_availability_set.web-avset /subscriptions/<your-subscription-id>/resourceGroups/test_rg/providers/Microsoft.Compute/availabilitySets/my-web-vm-av
terraform import azurerm_linux_virtual_machine.web-vm /subscriptions/<your-subscription-id>/resourceGroups/test_rg/providers/Microsoft.Compute/virtualMachines/my-web-vm
terraform import azurerm_managed_disk.web-datadisk /subscriptions/<your-subscription-id>/resourceGroups/test_rg/providers/Microsoft.Compute/disks/my-web-vm-datadisk
terraform import azurerm_network_interface.web-vm-nic /subscriptions/<your-subscription-id>/resourceGroups/test_rg/providers/Microsoft.Network/networkInterfaces/my-web-vm-nic
terraform import azurerm_virtual_machine_data_disk_attachment.web-diskattachment /subscriptions/<your-subscription-id>/resourceGroups/test_rg/providers/Microsoft.Compute/virtualMachines/my-web-vm/dataDisks/my-web-vm-datadisk

After running those, we can see that a terraform plan outputs no changes. Hurray!

Regenerating the state: using state manipulation

Let’s now see how we can achieve the same result using the Terraform state commands for our database virtual machine folder.
First of all we will need to copy our old state file from the 1-single_filei folder into our database folder.
Once this is done, running a terraform init and plan will output this:

# azurerm_availability_set.web-avset will be destroyed
# azurerm_linux_virtual_machine.web-vm will be destroyed
# azurerm_managed_disk.web-datadisk will be destroyed
# azurerm_network_interface.web-vm-nic will be destroyed
# azurerm_network_security_group.my-nsg will be destroyed
# azurerm_resource_group.resourcegroup will be destroyed
# azurerm_subnet.my_subnet will be destroyed
# azurerm_virtual_machine_data_disk_attachment.web-diskattachment will be destroyed
# azurerm_virtual_network.network will be destroyed
Plan: 0 to add, 0 to change, 9 to destroy.

We removed the code managing all those resources in this folder but the state file still keeps track of them so naturally Terraform thinks we want to delete them.
To solve this, we will use the terraform state rm command that removes resources from the state. The documentation is here : https://www.terraform.io/docs/cli/commands/state/rm.html
It’s quite simple here, just run a command for each resource that the plan tells us would be removed:

terraform state rm azurerm_availability_set.web-avset
terraform state rm azurerm_linux_virtual_machine.web-vm
terraform state rm azurerm_managed_disk.web-datadisk
terraform state rm azurerm_network_interface.web-vm-nic
terraform state rm azurerm_network_security_group.my-nsg
terraform state rm azurerm_resource_group.resourcegroup
terraform state rm azurerm_subnet.my_subnet
terraform state rm azurerm_virtual_machine_data_disk_attachment.web-diskattachment
terraform state rm azurerm_virtual_network.network

Our state file is now in sync with our code, and running a new Terraform plan outputs no changes!

This is the end of step 1, we now have separated our monolithic file into different folders that each have their state up to date. In the next step we’ll see how to share global settings between your folders. See you soon!