Terraform to Terragrunt: Migration Guide

In the fast-evolving landscape of cloud infrastructure, the importance of Infrastructure as Code (IAC) has become paramount. Among the various tools available, Terraform has emerged as a popular choice due to its versatility in managing infrastructure across different cloud providers. However, as projects scale and organizations adopt multi-environment setups across multiple AWS accounts, challenges begin to surface.

At the heart of these challenges is the cumbersome task of managing multiple state files, code repetition, and variable redundancy. The need to create resources one by one, coupled with the sheer volume of lines of code required, can make the IAC process intricate and error-prone. Faced with these obstacles, the quest for an optimized solution led us to explore Terragrunt.

Embracing Terragrunt for Enhanced IAC:

Terragrunt, an open-source wrapper for Terraform, proved to be the missing piece in our infrastructure puzzle. It addresses the pain points associated with managing Terraform configurations at scale. Let’s delve into some of the key benefits that Terragrunt brings to the table.

1. Simplified State Management:

Terragrunt simplifies the management of multiple state files, providing a clean and organized structure. With each module encapsulated within its own directory, state files are neatly compartmentalized, reducing the risk of conflicts and improving overall project maintainability.

2. Eliminating Code Repetition:

One of the primary challenges in Terraform is code repetition. Terragrunt introduces the concept of “remote” blocks, allowing shared configurations to be centralized. This means common configurations can be maintained in a single location, reducing redundancy and enhancing code readability.

3. Efficient Variable Handling:

Terragrunt’s ability to handle variables efficiently is a game-changer. It allows for the inheritance of variables across different environments, eliminating the need to redefine them repeatedly. This not only streamlines the code but also enhances consistency and reduces the likelihood of errors.

4. Enhanced Resource Management:

Unlike Terraform, Terragrunt enables the creation of entire components in a single operation, offering a more holistic approach to infrastructure provisioning. This feature significantly improves the efficiency of resource creation, making it easier to manage complex infrastructures with ease.

5. Sequential Deployment of Resources:

With Terragrunt, the process of creating resources is no longer limited to a linear fashion. The ability to create components one by one provides a more granular control over resource provisioning, facilitating better error handling and troubleshooting.

6. Optimized Performance:

Terragrunt’s optimized approach to managing Terraform configurations translates into improved performance. The modular and organized structure it introduces leads to faster execution times, making it an ideal choice for projects of all scales.

Terragrunt vs. Terraform Comparison

Terraform and Terragrunt are both tools used for Infrastructure as Code (IaC). Let’s take a look at their similarities and differences.

Below you can find Terraform vs. Terragrunt table comparison.

Modern cloud infrastructure demands robust and scalable solutions, and Terragrunt emerges as a key player in orchestrating Infrastructure as Code (IaC) with Terraform. In this comprehensive guide, we will explore the inner workings of Terragrunt, focusing on its core concepts and providing a detailed example based on a sophisticated directory structure.

Core Concepts of Terragrunt Configuration

Terragrunt Configuration File: `terragrunt.hcl`

Terragrunt relies on a configuration file named `terragrunt.hcl`, strategically placed in the root directory of Terraform projects or within specific module directories. This file serves as the epicenter for customizing Terragrunt’s behavior.

Local Variables Block

locals {
  region_vars = read_terragrunt_config(find_in_parent_folders("common-vars.hcl.json"))
}

The localsblock is where our journey begins. Here, we define a local variable `region_vars` using `read_terragrunt_config`. This nifty function reads configurations from “common-vars.hcl.json” in parent folders, enabling us to centralize and reuse common settings across our infrastructure.

Dynamic Provider Configuration

generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
provider "aws" {
  region  = "${local.region_vars.locals.aws_region}"
  profile = "${local.region_vars.locals.aws_profile}"
}
EOF
}

Terragrunt allows dynamic generation of provider configurations using the `generate` block. In this example, we create a “provider.tf” file dynamically, filling the AWS provider configuration with values from our `region_vars` local variable. This dynamic approach enhances flexibility and maintainability as configurations evolve.

Remote State Management

remote_state {
  backend = "s3"
  generate = {
    path     = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  config = {
    bucket          = "test-non-prod"
    key             = "${path_relative_to_include()}/terraform.tfstate"
    region          = "${local.region_vars.locals.aws_region}"
    encrypt         = true
    dynamodb_table  = "test-non-prod-lock-table"
  }
}

Managing Terraform state is simplified with the `remote_state` block. This block is a game-changer, streamlining the storage and retrieval of Terraform state files. Let’s break down its significance:

Dynamic Generation of “backend.tf”

generate = {
  path      = "backend.tf"
  if_exists = "overwrite_terragrunt"
}

The `generate` block ensures the dynamic creation of a “backend.tf” file, specifying the backend configuration. This file serves as the link between Terraform and the chosen backend, defining how state files are stored and accessed.

Centralized State File Directory Structure

key = "${path_relative_to_include()}/terraform.tfstate"

The `key` parameter within the `config` block plays a crucial role in structuring the remote state file directory. `path_relative_to_include()` dynamically captures the relative path of the current configuration file within the project, ensuring each module has its dedicated directory for storing state files.

Intelligent Use of Local Variables

region = "${local.region_vars.locals.aws_region}"

The `region_vars` local variable consolidates AWS region information, serving as a single source of truth for critical configuration details. This standardized approach enhances maintainability and ensures state files are stored in the correct AWS region.

Eliminating Multiple State File References

By adopting this approach, we eliminate the need for multiple state file references spread across different directories. Each configuration block contributes to a unified directory structure for state files, simplifying management and retrieval.

Crafting a Terragrunt Configuration: Real-world Example

Now, let’s apply these concepts to a real-world scenario with a complex directory structure.

Directory Structure

.
├── README.md
├── common-vars.hcl.json
├── non-prod
│   └── us-west-2
│       └── dev
│           ├── ecr
│           │   ├── custom-vars
│           │   │   └── ecr.hcl.json
│           │   └── terragrunt.hcl
│           ├── efs
│           │   ├── custom-vars
│           │   │   └── efs.hcl.json
│           │   └── terragrunt.hcl
│           └── ... (other modules)
├── prod
│   └── us-west-2
│       └── prod
│           ├── ecr
│           │   ├── custom-vars
│           │   │   └── ecr.hcl.json
│           │   └── terragrunt.hcl
│           ├── efs
│           │   ├── custom-vars
│           │   │   └── efs.hcl.json
│           │   └── terragrunt.hcl
│           └── ... (other modules)
└── terragrunt.hcl

Terragrunt Configuration File

terraform {
  source = "${local.base_source_url}"
}

include {
  path   = find_in_parent_folders()
  expose = true
}
locals {
  ecr          = read_terragrunt_config("custom-vars/ecr.hcl.json")
  global_vars  = read_terragrunt_config(find_in_parent_folders("common-vars.hcl.json"))
  env_vars     = read_terragrunt_config(find_in_parent_folders("env-vars.hcl.json"))
  base_source_url = "${gitlab-url}"
}
inputs = merge(
  local.ecr.locals,
  local.global_vars.locals,
  local.env_vars.locals,
  {}
)

This Terragrunt configuration file exemplifies a modular and extensible approach to AWS ECR deployment. Leveraging dynamic source URLs, inclusion magic, local variables, and input synthesis, it creates a symphony of flexibility and organization.

The `terragrunt.hcl` file is a configuration file used by Terragrunt, a tool designed to work in conjunction with Terraform to simplify and enhance the management of Terraform configurations. This file is typically located in the root directory of a Terraform project or within specific module directories. The `terragrunt.hcl` file contains settings and parameters that customize Terragrunt’s behavior for your project or module. Let’s break down the key sections of this file and understand their significance:

1. Dynamic Source URL with Terraform:

terraform {
  source = "${local.base_source_url}"
}

The `terraform` block specifies the source URL for Terraform modules. In this case, the `source` attribute is dynamically set using the `local.base_source_url` parameter. Dynamic source URLs enable flexible management of module dependencies and versions, accommodating changes and updates seamlessly.

2. Harnessing Terragrunt’s Inclusion Magic:

include {
  path   = find_in_parent_folders()
  expose = true
}

The `include` block utilizes Terragrunt’s inclusion mechanism. By including configurations from parent folders, Terragrunt encourages a modular and DRY (Don’t Repeat Yourself) approach. This strategic inclusion minimizes redundancy in infrastructure code, fostering a more organized and maintainable project structure.

3. Local Variables: The Backbone of Configuration Enrichment:

locals {
  ecr             = read_terragrunt_config("custom-vars/ecr.hcl.json")
  env_vars        = read_terragrunt_config(find_in_parent_folders("env-vars.hcl.json"))
  global_vars     = read_terragrunt_config(find_in_parent_folders("common-vars.hcl.json"))
  base_source_url = "${gitlab-url}"
}

The second `locals` block enriches the configuration with essential variables. Here, three local variables are defined:
- **`ecr`**: Fetches ECR-specific configuration details from “custom-vars/ecr.hcl.json.”

- **`env_var`**: Fetches env-specific configuration details from “env-vars.hcl.json.”
- **`global_vars`**: Gathers global variables from a common configuration file (“common-vars.hcl.json”).
- **`base_source_url`**: Establishes a flexible base URL for Terraform modules, facilitating modularization and version control.

4. Crafting a Comprehensive Inputs Configuration:

inputs = merge(
  local.ecr.locals,
  local.global_vars.locals,
  local.env_var.locals,
  {}
)

The `inputs` block acts as a synthesis point, seamlessly merging local variables with any additional inputs. This thoughtful amalgamation strategy creates a comprehensive set of parameters for Terraform modules, ensuring adaptability as the project evolves.

Examples of common and Env variables

Common Variable

{ 
 "locals": {
 "aws_profile": "aws-test",
 "Region": "${reverse(split(\"/\", get_original_terragrunt_dir()))[2]}",
 "Environment": "${reverse(split(\"/\", get_original_terragrunt_dir()))[1]}",
 "Component": "${reverse(split(\"/\", get_original_terragrunt_dir()))[0]}",
 "shared_vpc_id": "vpc-id",
 "Vertical": "operations",
 "Managed": "terragrunt"
}}

Env Variable

{ 
 "locals": {
 "ami": "ami-025d5160f68",
 "vpc-id": "vpc-037be9fe86",
 "Private_Zone": "zone-id",
 "Public_Zone": "zone-id",
 "volume_type": "gp3",
 "key_arn": "kms-arn",
 }
 }

The integration of Terragrunt into our Infrastructure as Code workflows brought about a remarkable reduction in code complexity. Previously, managing a large-scale infrastructure demanded around 20,000 lines of code, leading to challenges in code readability, maintainability, and increased likelihood of errors. Terragrunt’s modular structure and the elimination of code repetition drastically streamlined our codebase, reducing it to a mere 2,000 lines. Moreover, the burden of handling variables was similarly alleviated, with the number of variable lines plummeting from 50,000 to a concise 3,000. This reduction not only enhances code maintainability but also minimizes the risk of errors, making the entire IAC process more efficient and developer-friendly.

With these improvements, creating multiple infrastructures in one go became a reality. Terragrunt’s ability to handle resource dependencies seamlessly meant that we no longer needed to painstakingly manage multiple backend.tf files. The hassle of coordinating and maintaining dependencies across different components of our infrastructure was significantly mitigated, leading to a more streamlined and error-resistant deployment process.

In essence, the adoption of Terragrunt not only enhanced the clarity and efficiency of our Infrastructure as Code but also brought about a substantial reduction in code complexity, demonstrating its prowess in managing large-scale, multi-environment setups with finesse.

Conclusion

In conclusion, mastering Terragrunt involves understanding its core concepts and applying them to real-world scenarios. By leveraging local variables, dynamic provider configurations, and intelligent state management, you can streamline your Infrastructure as Code workflows. The provided example demonstrates the versatility and power of Terragrunt in managing complex AWS infrastructure. Embrace Terragrunt, and elevate your AWS infrastructure provisioning to new heights!

“If you’ve found this blog post helpful, consider buying me a coffee to support my writing!”

Acknowledgments
A sincere thank you to Abhishek Srivastava, Gaurav Sharma, Ankit Mehta and Parmjeet Malik for playing pivotal roles in guiding this migration. Abhishek’s leadership, combined with Gaurav , Ankit and Parmjeet’s invaluable support, enriched the content with their expertise. Special thanks to all for their collective efforts in making this blog a reality.