Terraform Provider Debugging: A step by step guide

Terraform Provider Debugging: A step by step guide

Terraform is HashiCorp’s infrastructure as code (IAC) tool. It lets you define resources and infrastructure in human-readable, declarative configuration files, and manages your infrastructure’s lifecycle. Terraform plugins let terraform interact with cloud platforms and other services via their APIs. These plugins are called Providers. Many providers have been written by HashiCorp and open-source communities.

While developing a provider (currently allowed only in Golang), I faced some challenges when I needed to debug the code. In this article, I want to share my experience about debugging process while developing and hope it would be helpful for everyone struggling with the same.

NOTE: here, we will be using IBM cloud terraform examples for references. I am using mac, so paths can vary based on OS.

So, let’s start.

The first thing to understand is that, terraform by default invokes the provider from HashiCorp’s default registry location. So, in following terraform config, where no specific registry path (private or custom) is mentioned, it is pointing to IBM cloud terraform provider from this location.

terraform {
required_providers {
ibm = {
source = “IBM-Cloud/ibm”
version = “1.46.0”
      }
  }
}
provider “ibm” {}

This is useful only when the provider has been tested thoroughly and pushed to registry. But this is not the case while the development is in progress. We developers need to test the provider while the changes are on our local environment. And it becomes tricky because providers are not executable binaries. I will discuss three strategies to test the things locally.

Strategy I:

Define the provider source in the terraform config: When we have a provider executable in good shape and we need to test it:

• Go to your IBM terraform provider directory and run “make build” command to install a binary. [Optionally, you can also build a binary using a build command, but recommended way is to disable all optimization and inlining.]

• This step will create a provider binary file in bin folder, but note that it is not executable, so you will receive an error if you try to run it directly:

• Terraform by default, looks up for the plugins in $HOME/.terraform.d/plugins (unix based systems) & %APPDATA%/.terraform.d/plugins (for windows) directory. We will try to build a path in the following pattern, so terraform can recognise it as per recommendation by terraform (https://developer.hashicorp.com/terraform/language/modules/sources#terraform-registry).

Here site URL can be any dummy URL, provider namespace reference can also be custom and version can also be user defined.

• Put the terraform binary in this path and it should look like this:

/Users/narinderkaur/.terraform.d/plugins/terraform.example.com/IBMCloud/ibm/1.44.0/darwin_amd64/ terraform-provider-ibm

• Now we are ready to test our binary, let us see how we will put this local provider in terraform config file:

terraform {
required_providers {
ibm = {
source = “terraform.example.com/IBMCloud/ibm”
version = “1.44.0”
      }
   }
}
provider “ibm” {}

With this config file, terraform would try to run IBM provider from your local.

Strategy II:

Override the provider source in the terraform CLI configuration file: In this second approach, we need to understand that terraform works on default configuration settings and in order to override these settings, terraform has given the users a way to define it in .terraformrc file (on Unix systems) and terraform.rc(on windows systems), whose default location is $HOME or %APPDATA% respectively. Using environment variable TF_CLI_CONFIG_FILE, these default locations of CLI configurations can also be altered.

The CLI configuration file contains a provider_installation block, which allows overriding Terraform’s default installation behaviour.

provider_installation {
dev_overrides {
“ibmcloud/ibm” = “/Users/narinderkaur/go/bin”
  }
}

The above configuration overrides the package for the hashicorp/ibm provider binary with /Users/narinderkaur/go/bin/terraform-provider-ibm. This disables the version and checksum verifications for this provider and forces Terraform to look for the IBM provider plugin in the given directory.

Now we are ready to test our binary, let us see how we will mention this local provider in terraform config file:

terraform {
required_providers {
ibm = {
source = “ ibmcloud/ibm”
      }
    }
}
provider “ibm” {}

The above two methods are almost similar, the only difference is that in second case, terraform would skip the version and checksum verification. And you will not require to run a terraform init command.

Strategy III

Running the provider in debug mode: Sometimes, we come across a bug, which we can debug only by understanding how the code is behaving at run time and how the state of code is changing. In such situation, above two methods are not so useful, and we need a real debugger. Now because terraform providers are not executable binaries, we cannot simply put them into the debugger mode and run. Let’s follow the below described steps to achieve this.

• The first step is to build the binary of terraform provider with debug mode. And in this step, it is also important to note that when debug mode is true, we need to add ProviderAddr also to ServeOpts. This property is kept only for debug mode, if not provided, by default the provider will run with the name of “provider”

• Once added the required code modification, now it is time to build. Run the command “make build”. If everything is good, it will generate the binary in your go path’s bin folder with the name of terraform-provider-ibm.

Next step is to mention the source of provider binary to terraform. We will not use anything new here, we will follow above mentioned strategy II and use provider_installation’s dev_overrides property to define the address for provider lookup.

• When we run directly terraform provider in debug mode, it does not give error as described in Strategy I(step II). It will run successfully and would provide a message asking you to set the TF_REATTACH_PROVIDERS environment variable to the value shown. This environment variable will instruct Terraform Core to not to start a new subprocess for the given provider address and attach to this existing plugin process, to send its requests to the socket address.

• For debugging, we can use two ways, one is to use delv (https://github.com/go-delve/delve). And other way is to use an ID. For simplicity, we will stick to debugging via Goland(go IDE by JetBrains). So now in Goland, we need to attach the debugger to a process, for that, go to Goland Menu->Run->Attach to Process.

• It will give you the list of running processes. Select the one from the list. You can use the process id to match from the previous output.

• You will see the output similar to following, informing about the server running in debug mode:

• Let us add some break points where ever we need to debug:

• As mentioned in step 4, create an environment variable TF_REATTACH_PROVIDERS as below:

• Now everything looks nice and it is good time to test your terraform configuration. Let us create one. As you can see below, now we used source attribute to point to the binary of the provider.

• Let’s execute it now. Also note that in this case, you don’t need to run “terraform init” command. We will directly invoke terraform plan and terraform apply. And Tada!!! We got to our breakpoint to dig deeper.

Conclusion: Through this blog, I have tried to explain various paths of terraform testing on local environment. Although, debug mode strategy is my favourite one, but because it is little more effort taking. If not required, one can use others ways to run the plugins from their local environments.