Better make for automation

Everyone probably knows about make and Makefiles. Initially a build automation tool, it’s often used as wrapper around different tools to automate repetitive actions. Also, shell scripts could be used for the same purpose.

However, this approach not very cross platform (different OS have different cli tools with different set of options) and it’s hard to create something sophisticated and resilient with it.

Fortunately, there are similar tools which use scripting languages like Python (invoke) and Ruby (rake). So you can write tasks using these languages, which will be platform-independent and could do anything language and its libraries support.

In this article I will focus on how it could help with different terraform tasks (terraform is an IaC tool). Since I do not use Ruby much, examples will use Invoke, but idea is the same.

Let's skip basic hello world example, it could be found in getting started guide. All you need to start:

• python 3.6+ for these examples
• install invoke (pip3 install invoke )
• create file tasks.py (code below)

As an example, let’s try to fix issues with terraform providers lock file, I described in another article, by creating terraform init invoke wrapper.

So, we have 2 issues:

• lock file contains only hash sums for OS you’re running terraform (and on others terraform init will fail)
• when you have TF_PLUGIN_CACHE_DIR env variable set, your lock file will be invalid (contain only h1 type hash sums). NOTE:TF_PLUGIN_CACHE_DIR enables caching for provider binaries and saves a lot of time and network traffic.

To fix first one, we need to run terraform providers lock -platform=os1 -platform=os2 after terraform init. And to avoid second one, TF_PLUGIN_CACHE_DIR should be unset for both terraform init and terraform providers lock.

terraform init wrapper logic

Code implementation:

Check code in Nklya/invoke-terraform, PRs: #1, #2

Now let’s extend this init wrapper by adding dynamic remote state configuration. Basically, you can achieve the same (DRY) with terragrunt, but since we already have invoke wrapper, let’s add it here instead of adding new tool.

By default, state backend configuration is static and looks like this:

terraform {
  backend "s3" {
    bucket         = "test-terraform-state"
    key            = "some/path"
    region         = "us-east-1"
  }
}

It’s ok if you have only couple of them, but it’s better to have more smaller states. And in this case static configuraion is quite tedious and error prone.

Since terraform init supports command line options to define backend configuration options, all we need is to pass them to the terraform init from previous example, snippet (full code in PR#3):

Basically step-by-step:

• first we calculate relative path to the root of repository to get key for state (“folder” in S3 bucket)
• then we load backend parameters from config.yaml in the root of the repo
• and run terraform init with all “backend-config” keys required
• all that required from terraform code — to have terraform {backend “s3” {}} section without parameters

In summary — yep, all of this could be done with some bash/makefile/powershell/you name it, but from my point of view this implementation with Invoke is repeatable in different environments, easy to read, understand and improve.

Code repository is Nklya/invoke-terraform, check next article with extended functionality of terraform wrapper.