4 Use Cases for the Terraform CDK

As someone who has written my fair share of TF, I’m a bit biased towards a declarative approach to IaC. A declarative approach shifts the maintenance burden onto the provider maintainers and enables practitioners who don’t know how to code to get started quicker.

That being said, I think most reasonable people would agree that not everything should be or can be defined in a static, declarative way. Terraform’s dynamic resource blocks, and for_each support can quickly leave you wanting for more. Let’s dive into the CDK and see some areas where it can help.

What is the Terraform CDK?

In 2019 AWS released their Cloud Development Kit (CDK), built upon the Construct library. The Construct library allows developers to define Javascript or Typescript-based classes that can be used to generate classes in any language that implements the JSii interface.

The Terraform CDK (TF CDK) is built on the same Construct and JSii libraries by automatically creating Constructs for Terraform Providers, Resources, and Data Sources. This allows a practitioner to leverage languages like Python, Typescript, Go, and more to develop language-native representations of Terraform resources.

The TF CDK is then responsible for taking the code written in one of these other languages and synthesizing it into a TF configuration in a JSON format. This is then handed off to the Terraform binary to be planned and applied.

You can think of the TF CDK as a language library for providers and its resources, as well as a pre-process step that turns any language into TF configuration.

Use Cases

Dynamic Resource Attributes

Code example show casing the issue with trying to use nested dynamic blocks with AWS’s WAF v2 Rule Group Statements which can be recursively nested

At some point, you’re going to want to base the attributes of your resources on a dynamic set of data. In the most straightforward cases, for_eachand dynamic{} blocks allow you to capture this relationship between data and your resources.

However, some resources, such as the AWS WAFv2 rule group, have more complex data structures, given their ability to support recursive statements. An example of this relationship is shown above on the right-hand side. Supporting this type of relationship with Terraform’s dynamicblocks results in convoluted code. It might not capture the entire data structure in the case of an unbounded recursion depth.

Contrast this to the Dynamic Resource Attributes Example on the left-hand side, which allows us to leverage Python’s native JSON parsing and convert these nested statements directly into the appropriate CDK classes.

Dynamic Module Composition

Hierarchical diagram showing how the App A module leverages children modules as building blocks

Terraform does not provide the ability to source Provider and Module versions dynamically as this information has to be known before initialization. In general, this is not an issue. Still, it usually means that you end up with a hierarchical structure to your modules where your outermost module represents a “release” or a specific configuration of all of its children modules.

This release concept is ideal as it provides us with point-in-time snapshots of module configurations that interact well with each other. However, what if you’re running a PaaS service and have one hundred customers with varying release versions. It might be beneficial to externalize the set of module versions a customer is running on. This allows us to support capabilities like feature flags without having to modify any TF code manually.

The CDK provides us with this ability because it acts as a TF code generation layer; we can easily pull our release information from any external system like a Database, REST API, Consul, or even from a file on disk.

The Dynamic Module Composition Example showcases and example of reading a set of module versions from a JSON file on disk. It then passes this information into the CDK library to generate the final TF code.

External Configuration Integration

Diagram that shows a Python process making a GET request to the BoredApi and using that response as input to the CDK.

Terraform has access to hundreds of providers which have varying degrees of data sources from which we can pull information. But, what if we wanted to base some of our TF configuration on data from Kafka, a REST API, or even a service that doesn’t have a provider?

Unfortunately, that isn’t possible without an external process pulling that data and translating it into TF variables before we run terraform apply. This is another example where the TF CDK acts as that external process that provides us with two possibilities.

1. Automatically reacting to events received, generating new TF code, and then executing it.
2. Pulling data from an external source and using it to dynamically generate our TF code.

The External Configuration Integration Example leverages scenario two by making a GET request to the BoredApi and then using that response to define the name of an AWS Compute Instance.

Post Provision Steps

Flow diagram showing a Python process, adding a resource to a CDK Stack, generating TF code, applying it adding a commit stage and repeating

I’ve only ever run into this issue once with Terraform, and it’s specifically related to how the Terraform Palo Alto provider works. Unfortunately, the Palo Alto provider cannot execute a commit at the end of an apply. As a result, all of the changes that have been updated in your Statefile are staged. This means that you need to have some process that can execute the commit against the Palo Alto API at the end of your terraform apply. It also means that if for some reason, the commit step fails, our TF state and the Palo Alto provider are essentially out of sync.

The Post Provision Steps Example leverages the ability for the TF CDK to generate TF code on demand and the ability to control the Terraform lifecycle by injecting a commit step after each resource is added.

Cons

I’m going to omit the pros section as the 4 use cases represent most of them already. Most of the cons relate to the maturity of the project, but I do want to call them out as they are important to understanding whether or not your team can support the CDK in its current state.

1. The CDK fetches modules, providers, and their JSii classes using the cdktf get command. It takes long enough to get a cup of coffee and, unfortunately, doesn’t seem to perform a diff before running; instead, it re-downloads everything each time. You probably won’t run it too much, but terraform init is an extremely quick operation that is functionally equivalent.
2. Import syntax highlighting and Intellisense are non-existent when it comes to the Python implementation of the CDK. I suspect this is because the CDK downloads the JSii classes into a non-standard import folder, but it compounds with the next issue.
3. No docs on the names of classes or their implementation mean that in the case of AWS, you’re going hunting through a 40MB Python file with no syntax highlighting using the find command to search for their implementation.
4. Class names are auto-generated and result in some interesting names. In the case of our recursive WAF Rule Statements from above, this is an example.Wafv2RuleGroupRuleStatementOrStatementStatementGeoMatchStatementForwardedIpConfig
5. As with all new projects, the error messages can get quite verbose and the CDK is no different. In the case below, I was passing in the wrong type of object when creating a TerraformHclModule

Stack trace example of passing in the wrong argument to a CDK class

Conclusion

The TF CDK is a great tool to add to your belt, but I would caution against switching over to it wholesale. While it can feel comfortable to use a language that you already know, declarative code has many unseen benefits when it comes to maintainability. Because we are simply declaring what type of resources we want to exist, the underlying implementation can easily be upgraded as long as it provides the same inputs and outputs. While the same could be said of the auto-generated JSii classes, you’re also dealing with two additional layers of abstraction that can change at any time.

Leverage the CDK for what it’s best at, a thin layer that sits on top of Terraform, allowing you to dynamically grab information and then use that to generate the underlying TF code. More specifically, I think there is a sweet spot where ninety-five percent of your TF is declarative HCL, and the remaining five percent is some variation of CDK code loading data and utilizing the existing TF modules. This allows teams to maintain a shared set of modules while still allowing each team to maintain a light customization layer on top.

I enjoyed exploring the TF CDK, and I’m excited to see how it changes as the project continues to iterate. Hopefully, you learned something from this post, however, I’m also curious what your experience has been or if you have any other ideas on how to leverage the CDK. Reach out to me on Twitter; I would love to continue the conversation.

GitHub - peytoncasper/tf-cdk-examples