Building Terraform from Source with a Patch

Dirk Avery
Jun 18 · 9 min read

On occasion, you need to build Terraform from source. For example, you might need to try out a patch that hasn’t made it into the main repository yet. This is how you do it.

(Photo credit: David McBee)

NOTE: This is a modified version of my previous article.

If you’re interested in workflows using Terraform, check out my article on secure and simple Terraform.

Objectives

  • We are going to build Terraform from source
  • We are going to build the AWS Terraform provider from source
  • We are going to modify Terraform and the AWS Terraform provider to use a patch I wrote to fix a bug when you get AWS credentials

When we’re done, you’ll have two binary files: Terraform and the AWS provider plugin. These will both use my patch so that you can test whether the patch fixes your issues and if the patch introduces any new issues. Thank you for your testing help!

Both Terraform and the AWS provider need to be built to ensure they’re compatible with each other and since both use the patch.

NOTE: Which directory you are in makes a big difference with these instructions. Each step leaves you in the right place for the next step. If you’re jumping in and out, make sure you know what you’re doing.

1. Install Git and Go

You’re probably covered here but if not, check out these instructions.

2. Uninstall Terraform

Since we’re building a new Terraform, make sure that earlier installs aren’t lurking about.

One of these commands should uninstall Terraform for most people:

$ brew uninstall terraform
Uninstalling /usr/local/Cellar/terraform/0.11.11... (6 files, 102.3MB)
$ rm $GOPATH/bin/terraform
$ rm /usr/local/bin/terraform
$ hash -r

Note that hash -r doesn’t uninstall anything but clears the Bash cache which might mess you up later.

Now when you try to check the Terraform version, you should get a not found error:

$ terraform --version
-bash: terraform: command not found

3. Make a home for Terraform

Terraform will need to be in the right place in your GOPATH. Start by making sure some key directories exist and get to the right place:

$ mkdir -p $GOPATH/src/github.com/hashicorp
$ cd $GOPATH/src/github.com/hashicorp

4. Git Terraform

We’ll first get Terraform in all its glory. Then, depending on your needs, in the next step you can get different versions of Terraform.

Assuming you do not have a fork, clone the main repository to your local machine:

$ git clone https://github.com/hashicorp/terraform.git
Cloning into 'terraform'...
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 203864 (delta 2), reused 2 (delta 2), pack-reused 203855
Receiving objects: 100% (203864/203864), 153.72 MiB | 20.75 MiB/s, done.
Resolving deltas: 100% (123318/123318), done.
Checking out files: 100% (5912/5912), done.
$ cd terraform
$ git fetch --all
* [new tag] v0.12.0 -> v0.12.0
* [new tag] v0.12.0-dev20190520H16 -> v0.12.0-dev20190520H16
* [new tag] v0.12.0-rc1 -> v0.12.0-rc1
* [new tag] v0.12.1 -> v0.12.1
* [new tag] v0.12.2 -> v0.12.2
* [new tag] v0.11.12 -> v0.11.12
* [new tag] v0.11.12-beta1 -> v0.11.12-beta1
* [new tag] v0.11.13 -> v0.11.13
* [new tag] v0.11.14 -> v0.11.14

5. Pick a version

If necessary, you can pick a specific version of Terraform. Otherwise, you’ll have the latest, bleeding-edge version and can skip this step.

$ git checkout tags/v0.12.0
Note: checking out 'tags/v0.12.0'.
You are in 'detached HEAD' state. You can look around, make
experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example: git checkout -b <new-branch-name>HEAD is now at 08918475d v0.12.0

In the above command, you can replace v0.12.0 with another version. To see all the available versions, go here.

6. Modify go.mod for Terraform

This is the magic step where we’ll get Terraform to use the patch. Here we’re using my credential-related patch but this could be used to test any other patch if you update the instructions appropriately.

Using a code or text editor, open up the go.mod file. The file is in the root directory of Terraform, which is where you were at at the end of the previous step ($GOPATH/src/github.com/hashicorp/terraform).

Add this line to the end of the file, after the closing parenthesis.

replace github.com/hashicorp/aws-sdk-go-base => github.com/YakDriver/aws-sdk-go-base v0.0.0-20190503174753-82bd97734e8f

If I’ve rebased the pull request, the hash (82bd97734e8f) might have changed. Go here to find the updated value.

7. Enable Go modules

NOTE: Earlier versions of Terraform are not compatible with Go modules. If you go too far back with the Terraform version, it’s possible you won’t be able to get anything to work. An easy giveaway indicating you’ve gone back too far is the disappearance of the go.mod and go.sum files. Try a more recent version!

Enable Go modules with the GO111MODULE environment variable:

$ export GO111MODULE=on

8. Build Terraform?

No, not yet. This is not the best time to build Terraform. When we work on the Terraform AWS provider, it is going to mess with our dependencies and will often introduce incompatibilities.

To avoid that, we will wait until after the AWS provider has had it’s fun with the dependencies. Then we’ll build both Terraform and the AWS provider with the same dependencies in place. Hopefully that way we’ll get a Terraform and AWS provider that are compatible with each other.

Not all versions of Terraform are compatible with all versions of the AWS provider. Matching up versions is quite forgiving, but if you build a very old Terraform with a very new AWS, or vice versa, you may run into problems.

9. Grab the commit hash

Before moving to the Terraform AWS provider, we are going to need a piece of information: the commit hash for Terraform. We’ll use the hash to make the Terraform AWS provider compatible with our Terraform.

$ git rev-parse --short HEAD
16823f43d

Keep track of this! You’ll need it later.

10. Make a home for the AWS provider

Like Terraform core, the AWS provider needs a directory.

$ mkdir -p $GOPATH/src/github.com/terraform-providers
$ cd $GOPATH/src/github.com/terraform-providers

11. Git the Terraform AWS provider

Grab the latest, bleeding-edge, development version of the Terraform AWS provider. In the next step, you can go back in time to an earlier version if necessary but this step is required either way.

Assuming you do not have your own fork, clone the main repository to your local machine:

$ git clone https://github.com/terraform-providers/terraform-provider-aws.git
Cloning into 'terraform'...
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 203864 (delta 2), reused 2 (delta 2), pack-reused 203855
Receiving objects: 100% (203864/203864), 153.72 MiB | 20.75 MiB/s, done.
Resolving deltas: 100% (123318/123318), done.
Checking out files: 100% (5912/5912), done.
$ cd terraform-provider-aws
$ git fetch --all
Fetching origin
Fetching upstream
remote: Enumerating objects: 66, done.
remote: Counting objects: 100% (66/66), done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 126 (delta 57), reused 56 (delta 56), pack-reused 60
Receiving objects: 100% (126/126), 62.64 KiB | 8.95 MiB/s, done.
Resolving deltas: 100% (74/74), completed with 28 local objects.

12. Pick an AWS provider version

If you need a particular version of the AWS provider, you can use Git to check it out. Otherwise, you’ll have the latest, bleeding-edge version and can skip this step.

$ git checkout tags/v2.15.0
Note: checking out 'tags/v2.15.0'.
You are in 'detached HEAD' state. You can look around, make
experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example:git checkout -b <new-branch-name>HEAD is now at 1a78b50db v2.15.0

You can see all the other available AWS provider versions here.

13. Get Terraform dependencies for AWS provider

Using Go modules and the commit hash from step 9 above, update the dependency versions:

$ go get -u github.com/hashicorp/terraform@16823f43d
go: finding github.com/hashicorp/terraform 16823f43d
go: finding github.com/gophercloud/gophercloud latest
go: finding github.com/hashicorp/go-plugin latest
go: finding github.com/antchfx/xpath latest
go: finding github.com/agl/ed25519 latest
go: finding github.com/kardianos/osext latest
go: finding github.com/mitchellh/go-linereader latest
...

I’ve seen some errors doing this but these haven’t affected what I’m testing. I continue without worrying about them. For example, these errors:

go: sourcegraph.com/sourcegraph/go-diff@v0.5.1: parsing go.mod: unexpected module path "github.com/sourcegraph/go-diff"
go: finding github.com/coreos/etcd v3.3.11+incompatible
go get: error loading module requirements

Checking the go.mod file, we’ll see Terraform has been updated:

$ cat go.mod | grep "terraform v"
github.com/hashicorp/terraform v0.12.0-alpha4.0.xx-16823f43deec

14. Modify go.mod for AWS provider

NOTE: This is just like a previous step but not a duplicate. Both are important!

Like we did earlier for Terraform, in this step we’ll get the AWS provider to use the patch. Here we’re using my credential-related patch but this could be used to test any other patch if you update the instructions appropriately.

Using a code or text editor, open up the go.mod file. The file is in the root directory of the AWS provider, which is where you were at at the end of the previous step ($GOPATH/src/github.com/terraform-providers/terraform-provider-aws).

Add this line to the end of the file, after the closing parenthesis.

replace github.com/hashicorp/aws-sdk-go-base => github.com/YakDriver/aws-sdk-go-base v0.0.0-20190503174753-82bd97734e8f

If I’ve rebased the pull request, the hash (82bd97734e8f) might have changed. Go here to find the updated value.

15. Build the Terraform AWS provider

Then the provider should build without error:

$ go build

16. Build Terraform

We’re now ready to build Terraform. Using Terraform’s included Makefile, this command will compile Terraform and install it into the Go bin folder.

$ cd $GOPATH/src/github.com/hashicorp/terraform
$ make dev

==> Checking that code complies with gofmt requirements...
GO111MODULE=off go get -u golang.org/x/tools/cmd/stringer
GO111MODULE=off go get -u golang.org/x/tools/cmd/cover
GO111MODULE=off go get -u github.com/golang/mock/mockgen
GOFLAGS=-mod=vendor go generate ./...
2019/02/25 16:56:34 Generated command/internal_plugin_list.go
# go fmt doesn't support -mod=vendor but it still wants to populate
# the module cache with everything in go.mod even though formatting
# requires no dependencies, and so we're disabling modules mode for
# this right now until the "go fmt" behavior is rationalized to
# either support the -mod= argument or _not_ try to install things.
GO111MODULE=off go fmt command/internal_plugin_list.go > /dev/null
go install -mod=vendor .
$ terraform -v
Terraform v0.12.0-dev

17. Install the AWS provider

We’ll put the newly minted provider in a special place where Terraform will find it. Also, we’ll give it a nonexistent but recognizable version number, 4.8:

$ mkdir -p ~/.terraform.d/plugins
$ cd $GOPATH/src/github.com/terraform-providers/terraform-provider-aws
$ mv terraform-provider-aws ~/.terraform.d/plugins/terraform-provider-aws_v4.8.15

18. Make HCL to test the problem

You should now be able to try your HCL against a working Terraform which uses the patch.

If you have an AWS profile that uses a credential process, you can verify that the patch is in use. Without the patch, assuming a role and using a credential process will fail (either one alone will work, just not together).

provider "aws" {
profile = "credproc"
region = "us-east-1"
assume_role {
role_arn = "arn:aws:iam::${var.account_id}:role/${var.role}"
session_name = "yo-session"
external_id = "23be7ac0-9dee-5681-b949-eaa5468c2f99"
}
}

19. Create a credential process profile

Creating a credential process profile is easy. The process called must return a JSON with valid credentials. Add the profile to your ~/.aws/credentials file:

[credproc]
credential_process = /some/process/that/returns/creds

20. Run Terraform

Initialize Terraform and look for the special version you gave it so that you’re sure that you’re not using the public binary:

$ terraform initInitializing provider plugins...The following providers do not have any version constraints in configuration, so the latest version was installed.To prevent automatic upgrades to new major versions that may contain breaking changes, it is recommended to add version = "..." constraints to the corresponding provider blocks in configuration, with the constraint strings suggested below.* provider.aws: version = "~> 4.8"Terraform has been successfully initialized!
...
$ terraform apply
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

21. Post your results

Please share whether your scenario worked or not and if any new issues came up! For the credential problems, here is the issue on GitHub.

Faun

The Must-Read Publication for Aspiring Developers & DevOps Enthusiasts

Dirk Avery

Written by

Cloud developer, AI buff, patent attorney, fan of cronuts. AWS Certified Solutions Architect — Professional. Go, Python, Bash. https://www.plus3it.com

Faun

Faun

The Must-Read Publication for Aspiring Developers & DevOps Enthusiasts