Direktiv: building a machine on AWS using Terraform WITHOUT a Terraform environment? (part 1)

Terraform & Direktiv working together

First question is how?

Direktiv is a serverless workflow engine, which *technically* means it spins up containers on-demand (using Kubernetes and Knative) and shuts them down in the sequence you’ve defined (workflow). So in the case of Terraform (or Ansible, shown in a different article) — we’re simply spinning up the latest version of the providers software as a container!

Second question is why?

1. Terraform and Ansible are GREAT solutions for infrastructure provisioning and management, with a heap of plugins and extensions. Why would we try and reinvent the wheel — especially if you consider how widely both have been adopted. Direktiv uses the plugins, extension and playbooks already built for both products and extends it into the broader microservices workflow architecture we provide
2. You get both environments without the effort of installing or maintaining them (or the resource cost of running them continuously). Just update the Docker build file with the latest (or version you’re comfortable with) if you don’t want to use the version we provide. This is the ephemeral nature of Direktiv workflows!

Btw, same applies for Ansible plugin and Docker build file.

You can also do this yourself! Install the Direktiv-all-in-one and play along!!

The Terraform Script

In this article we’re going to be looking at an implementation of a Terraform script. Firstly, we’re going to provision a virtual machine (Linux) to Amazon Web Services using the Terraform script below:

Terraform script used to deploy Linux machine to AWS

Note: examples for GCP and Azure here

The Terraform script above will deploy an Ubuntu machine to an AWS region. We will pass a couple of configuration parameters via the Direktiv workflow (region, secret and key).

The Direktiv Workflow

Now let’s have a look at the workflow YAML. We’re going to break this workflow down state-by-state.

State-by-state breakdown

Getting the Terraform script (Git)

Let’s use the Git plugin to get the Terraform script from GitHub. Let’s look at the workflow extract below:

So this is the breakdown of what’s happening. The Git plugin is cloning our Terraform example repository. The following argument to the git command is very important:

$out/instance/terraform-git-var

• The plugin is writing the output of the git clone (a tar.gz file) to an internal Direktiv variable called terraform-git-var.
• The /instance/ component of the argument instructs Direktiv to only create this variable for the workflow instance at runtime (which means the value is only available for this specific workflow and only created at the time when it runs — more about variables here).

Effectively we now have the full Git repository in a variable as a binary tar.gz file!

Running Terraform script

Ok … here we go! Below is the snippet which calls Terraform:

Terraform snippet running the script

Ok, so we have the full binary repository containing the Terraform script which we want to use (located in the /amazon directory) in a variable called terraform-git-var.

So to run the Terraform script we defined the function called tfrun. tfrun takes a couple of inputs:

• execution-folder: terraform-git-var/amazon: this is a location pointing to the Git repository (variable + the Git directory structure where the main.tf file is located)
• action: jq(.action): the action Terraform needs to apply to the script, passed as a JSON input formatted as follow:

{
   "action": "apply"
}

• args-on-init: ["-backend-config=address=http://localhost:8001/terraform-amazon-instance: we’re asking Terraform to pass the state output of the script back to the container (“plugin”) and store it in a variable called terraform-amazon-instance — to be used at a later stage or in different workflows (more on this later)
• variables: these are variables used by the Terraform main.tf script. The most import part to this is that the state-name needs to match up to the previous value of the args-on-init value.

Now let’s look at the function definition for tfrun:

• image: instruct Direktiv to use the vorteil/terraform:v1 container as the plugin or extension
• files: this declaration is documented here. This feature of Direktiv instructs the function to take the terraform-git-varvariable (key: terraform-git-var), defined as an instance scoped variable (scope: instance) and unzip it as a tar.gz file type!

It’s VERY important to understand the importance of the above — it means that the file inside the variable become available as a mounted & shared drive to the container! It also allows the execution-folder argument for the container to read the main.tf file from the directory.

Cleaning up and notify

The final 2 states checks the action which was sent to the Terraform script (either apply or destroy). If it is apply a message is sent to Discord with the machine details.

Final workflow

Here is a video of the workflow in action:

And here is the final workflow:

As always — feedback and questions welcomed!

PS: someone asked to see the output of the workflow. Here is the “apply” action output:

[10:31:48.257] Preparing workflow triggered by API. 
[10:31:48.356] Running state logic -- clone-terraform-examples:1 (action) 
[10:31:48.431] Sleeping until function 'git-command' returns. 
[10:31:49.991] running command 0 'clone https://github.com/vorteil/terraform-examples.git /mnt/shared/1uxr6qlijtrPRTCPEE9nwRJ9LyO/out/instance/terraform-git-var' 
[10:31:50.754] Function 'git-command' returned. 
[10:31:50.762] Transitioning to next state: deploy-aws (2). 
[10:31:50.769] Running state logic -- deploy-aws:2 (action) 
[10:31:50.769] {
  "action": "apply",
  "return": {
    "cmd0": {
      "cmd": "clone https://github.com/vorteil/terraform-examples.git /mnt/shared/1uxr6qlijtrPRTCPEE9nwRJ9LyO/out/instance/terraform-git-var",
      "output": "/mnt/shared/1uxr6qlijtrPRTCPEE9nwRJ9LyO/out/instance/terraform-git-var"
    }
  }
} 
[10:31:50.769] Decrypting secrets. 
[10:31:50.782] Sleeping until function 'tfrun' returns. 
[10:31:53.533] adding to the global map to control action ids 
[10:31:53.628] Checking if tfstate service http backend is alive... 
[10:31:53.706] Wait till backend service is functional 
[10:31:53.776] Initializing terraform.... 
[10:31:53.840] Reading in TFVars.json... 
[10:31:53.180] 
Initializing the backend...
 
[10:31:53.186] Fetching tfstate variable... 
[10:31:53.190] 
Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.
2021/07/07 00:31:53 [DEBUG] GET http://localhost:8001/terraform-amazon-instance
 
[10:31:53.214] 
Initializing provider plugins...
- Finding latest version of hashicorp/aws...
 
[10:31:54.625] - Installing hashicorp/aws v3.48.0...
 
[10:32:01.466] - Installed hashicorp/aws v3.48.0 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
 
[10:32:01.474] 
Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
 
[10:32:01.479] Executing 'apply' for terraform 
[10:32:02.152] 2021/07/07 00:32:02 [DEBUG] GET http://localhost:8001/terraform-amazon-instance
 
[10:32:02.157] Fetching tfstate variable... 
[10:32:09.194] aws_instance.web: Creating...
 
[10:32:19.161] aws_instance.web: Still creating... [10s elapsed]
 
[10:32:21.572] aws_instance.web: Creation complete after 13s [id=i-0382710118e5f441d]
 
[10:32:21.588] 2021/07/07 00:32:21 [DEBUG] POST http://localhost:8001/terraform-amazon-instance
 
[10:32:21.592] Saving new tfstate variable... 
[10:32:21.617] ╷
│ Warning: Value for undeclared variable
│ 
│ The root module does not declare a variable named "state-name" but a value
│ was found in file "terraform.tfvars.json". If you meant to use this value,
│ add a "variable" block to the configuration.
│ 
│ To silence these warnings, use TF_VAR_... environment variables to provide
│ certain "global" settings to all configurations in your organization. To
│ reduce the verbosity of these warnings, use the -compact-warnings option.
╵

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:
ip-address = "3.26.57.150"
 
[10:32:21.632] Sending output back to direktiv... 
[10:32:22.320] Fetching tfstate variable... 
[10:32:22.359] Fetching tfstate variable... 
[10:32:22.385] Function 'tfrun' returned. 
[10:32:22.386] Transforming state data. 
[10:32:22.393] Transitioning to next state: check_apply_or_destroy (3). 
[10:32:22.399] Running state logic -- check_apply_or_destroy:3 (switch) 
[10:32:22.399] {
  "action": "apply",
  "amazon_ip": "3.26.57.150"
} 
[10:32:22.399] Switch condition 0 succeeded 
[10:32:22.403] Transitioning to next state: send_message (4). 
[10:32:22.410] Running state logic -- send_message:4 (action) 
[10:32:22.410] Decrypting secrets. 
[10:32:22.421] Sleeping until function 'discordmsg' returns. 
[10:32:24.578] Function 'discordmsg' returned. 
[10:32:24.585] Workflow completed.