Creating dependencies between your Terraform modules

Terraform is an open-source software tool for building, changing and versioning your infrastructure. It manages your infrastructure changes based on your configuration, by updating it from one state to a new state. Terraform produces an execution plan describing what resources will change. Those resources are created, updated or deleted in parallel. While resources can be created independently, they often depend on other resources. To do so, Terraform relies on a built-in mechanism called resource dependencies to create them in the expected order. This mechanism is not built-in for Terraform modules as of version 0.12. However, with some key concepts, it is possible to force resources from different modules to be created in a specific order, making dependencies between modules possible. In this article, you will learn how to work around this limitation by using existing Terraform functionalities to create module dependencies efficiently.

Note: find the original article at https://www.fourcast.io/blog/creating-dependencies-between-your-terraform-modules.

Resources dependencies

Terraform creates resources in parallel and can automatically handle resources dependencies in most cases using implicit dependencies. If not specified, Terraform can’t know if a resource must be created before or after another one if there is no clear relationship within the Terraform code itself. To create a relationship between multiple resources, we must always use implicit or explicit dependencies between resources.

For example, let’s assume we want to create a Google Project within a folder. Here is what the Terraform code would look like without any relationship in place.

As the writer of the code, we know we must first create the folder in order to create the project within that folder. But there is no explicit nor implicit relationship for Terraform between the two resources. As a result, Terraform will try to create both resources at the same time and this will result in an error.

To prevent this from happening, we should create an implicit dependency:

The dependency is called implicit because Terraform can automatically find references of the object, and create an implicit ordering requirement between the two resources.

Sometimes, a resource can depend on another one but not reference it. In this case, we can create an explicit dependency using the depends_on key in a Terraform resource. Explicitly specifying a dependency is only necessary when a resource relies on some other resource’s behavior but doesn’t access any of that resource’s data in its arguments.

Module dependencies

Working with input and output arguments

Unlike resources, there is no built-in mechanism to create module dependencies as of Terraform 0.12. This means it is not possible to create an entire module only after another module has been created.

Hashicorp is currently working on the next version of Terraform (0.13), which will include module dependency as a built-in mechanism. The difference with the concept we are describing in this article is that we are only creating dependencies for resources depending on resources from a different module, instead of waiting for Terraform to apply changes on all resources in one module before applying the next module. Thus, it might often be faster to create resources dependencies between modules rather than have an entire module depend on another.

The solution relies on a combination of input/output arguments and resources dependencies. By creating a relationship between resources from different modules, we can force Terraform to create some resources in a specific order.

It is always possible to create all resources in a single module and use implicit or explicit dependencies to solve the problem. This approach however has some drawbacks, because you lose the advantages of using multiple reusable modules. Instead, we prefer to create dependencies between resources in multiple modules.

Let’s explain this with an example. This example is coming from our series on Setting up secure multi-cluster CI/CD pipelines with Spinnaker on Google Cloud Platform using Terraform. Read the full article here.

In our setup, we want to create a Kubernetes namespace for our DevOps operations. This namespace is defined in the cicd module. To be able to create the namespace, we first need to have a cluster available, which is created in the gke module. In addition, the cluster can only be created once the project is created, and this project is defined in gcp_project module.

To create the relationship, we are going to export some arguments from a module as output values (called outputs) and use them as input values (called variables) in another module.

From our gcp_project, we export the project_id and the vpc_name. They can only be available after both the project and the VPC network are created

In the gke module, we define both variables as inputs:

Creating the module dependency

Now, we are ready to create our first cross-module dependency! We first load the VPC resources with a data source. Because of the implicit dependency, this will only be loaded after the VPC network has been created. The cluster can then be created in the subnetwork we create in the gke module.

We now know that our cluster is only created after the project has been created. In fact, we have declared other implicit dependencies here: the cluster doesn’t only depend on the project or the VPC network, but also on the subnetwork and a BigQuery dataset. Those resources are part of the infrastructure we want to create.

We still need to create our Kubernetes resources in the cluster. To do so, we can export some cluster attributes from the gke module and use them as input in the cicd module. In this case, we will take advantage of the Kubernetes provider definition to create a dependency with all resources using it.

We export the following objects from the gke module first.

We can then configure the Kubernetes provider to use this cluster. The provider configuration will be passed implicitly to the module cicd. The Kubernetes resources will then be created in our cluster:

Conclusion

Even without a built-in mechanism to create dependencies between modules in Terraform, it is possible to make use of resource dependencies combined with input/output variables to create module dependencies for specific resources. Once Terraform 0.13 is released, module dependencies should be a built-in mechanism, but it can be slower than the method described in this article — depending on the structure of your modules.

Originally published at https://www.fourcast.io.