Introduction to Terraform Skeleton

Over and over, I had to face the same question: “How should I sparse my terraform code to make the most out of it?”.

Whether you are a newbie or an experienced CloudOps engineer, you’ll immediately agree that having a method or rather a methodology is keystone to ease the learning curve as well as speed up your game.

In this post, I expose a methodology as well as a template to: create, manage and expand your terraform projects.

It’s all in the modules: terraform modules is the basic building block when dealing with IaC. Although there are other areas of interest such as environments and components.

To exemplify, let’s assume we want to manage an organization called acme and that we need to cope four different components:

• iam: holds information and definition for role and service account management. Ex, implementation of a RACI matrix.
• compute: hosts definition for virtual machines and compute power related tasks.
• networking: supports network definition, subnetworks as well as firewalling.
• storage: controls definition and behavior for storage related tasks.

To create an empty folder structure, we provide a terraformer.py script that, when executed as follows, automatically populates this structure:

#python terraformer.py --project acme --subsystems compute iam networking storage --environments dev prod
#tree acme
acme/
├── compute
│   ├── environments
│   │   ├── dev
│   │   │   ├── main.tf
│   │   │   ├── outputs.tf
│   │   │   ├── terraform.tfvars
│   │   │   └── variables.tf
│   │   └── prod
│   │       ├── main.tf
│   │       ├── outputs.tf
│   │       ├── terraform.tfvars
│   │       └── variables.tf
│   ├── modules
│   │   └── README.md
│   └── shared
│       └── remotes.tf
├── iam
│   ├── environments
│   │   ├── dev
│   │   │   ├── main.tf
│   │   │   ├── outputs.tf
│   │   │   ├── terraform.tfvars
│   │   │   └── variables.tf
│   │   └── prod
│   │       ├── main.tf
│   │       ├── outputs.tf
│   │       ├── terraform.tfvars
│   │       └── variables.tf
│   ├── modules
│   │   └── README.md
│   └── shared
│       └── remotes.tf
├── modules
│   └── README.md
├── networking
│   ├── environments
│   │   ├── dev
│   │   │   ├── main.tf
│   │   │   ├── outputs.tf
│   │   │   ├── terraform.tfvars
│   │   │   └── variables.tf
│   │   └── prod
│   │       ├── main.tf
│   │       ├── outputs.tf
│   │       ├── terraform.tfvars
│   │       └── variables.tf
│   ├── modules
│   │   └── README.md
│   └── shared
│       └── remotes.tf
└── storage
    ├── environments
    │   ├── dev
    │   │   ├── main.tf
    │   │   ├── outputs.tf
    │   │   ├── terraform.tfvars
    │   │   └── variables.tf
    │   └── prod
    │       ├── main.tf
    │       ├── outputs.tf
    │       ├── terraform.tfvars
    │       └── variables.tf
    ├── modules
    │   └── README.md
    └── shared
        └── remotes.tf

The idea behind the automated structure is to max out reusability of components while preserving versatility of environments and components alike.

• acme/modules: hosts project wide terraform modules to be reused across the board.
• acme/{component}/modules: holds terraform component wide modules.
• acme/{component}/{environment}: contains definition and instantiation of such modules for any given environment.

We have intentionally left an example under our repository code to bring up a basic infrastructure containing a VPC, one subnetwork, a service account, a bucket, as well as a VM, based from debian.

When it comes to external modules, there are several ways to populate them:

• Host them in a separate git repository. This would allow you to initialize the acme/modules folder pointing to your existing repository as a git submodule.
• Host them in a shared file system. Allowing you to symlink your acme/modules folder to it.
• Host them as part of a bigger project in a git repository (our case). In this fashion, you can populate the modules for acme by pulling them from the external source. To achieve this, you can use the svn primitives (svn ls and svn export) supported by your git repository (in our case our GitHub branch).

If you look at GitHub you can check that the modules for acme are pulled from our git repository by using the following snippet of code (which combines svn ls and svn export).

#cd acme/modules
#for dir in $(svn ls https://github.com/globant/gcp-sandbox/branches/wip_terraformer/terraform-modules/); do svn export https://github.com/globant/gcp-sandbox/branches/wip_terraformer/terraform-modules/$dir; done

A full representation of acme project is presented below:

acme/
├── compute
│   ├── environments
│   │   ├── dev
│   │   │   ├── main.tf
│   │   │   ├── outputs.tf
│   │   │   ├── remotes.tf
│   │   │   ├── terraform.tfvars
│   │   │   └── variables.tf
│   │   └── prod
│   │       ├── main.tf
│   │       ├── outputs.tf
│   │       ├── remotes.tf
│   │       ├── terraform.tfvars
│   │       └── variables.tf
│   ├── modules
│   │   └── README.md
│   └── shared
│       └── remotes.tf
├── iam
│   ├── environments
│   │   ├── dev
│   │   │   ├── main.tf
│   │   │   ├── outputs.tf
│   │   │   ├── terraform.tfvars
│   │   │   └── variables.tf
│   │   └── prod
│   │       ├── main.tf
│   │       ├── outputs.tf
│   │       ├── terraform.tfvars
│   │       └── variables.tf
│   ├── modules
│   │   └── README.md
│   └── shared
│       └── remotes.tf
├── modules
│   ├── README.md
│   ├── gcs_bucket
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
│   ├── gcs_compute
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
│   ├── gcs_service_account
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
│   ├── gcs_subnet
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
│   └── gcs_vpc
│       ├── main.tf
│       ├── outputs.tf
│       └── variables.tf
├── networking
│   ├── environments
│   │   ├── dev
│   │   │   ├── main.tf
│   │   │   ├── output.tf
│   │   │   ├── terraform.tfvars
│   │   │   └── variables.tf
│   │   └── prod
│   │       ├── main.tf
│   │       ├── output.tf
│   │       ├── terraform.tfvars
│   │       └── variables.tf
│   ├── modules
│   │   └── README.md
│   └── shared
│       └── remotes.tf
└── storage
    ├── environments
    │   ├── dev
    │   │   ├── main.tf
    │   │   ├── output.tf
    │   │   ├── terraform.tfvars
    │   │   └── variables.tf
    │   └── prod
    │       ├── main.tf
    │       ├── output.tf
    │       ├── terraform.tfvars
    │       └── variables.tf
    ├── modules
    │   └── README.md
    └── shared
        └── remotes.tf

When you deep dive into any of the acme/{component}/{environment}/main.tf you can see that it holds the instantiation information to an existing module which when used with terraform.tfvars can be extremely flexible.

...
provider "google" {
...
}
terraform {
  backend "gcs" {
  ...
  }
}
module "default_vm" {
  source             = "../../../modules/gcs_compute"
  ...
}

Conclusion

In conclusion, we are showcasing a way to initialize new, or future, terraform projects while at the same time making room for refactoring existing ones. Leveraging on the common terraform modules structure is key to enable reusability across multiple organizations and even multi cloud providers.