Terraform Patterns, Observed
Part 5: Logic in Terraform
Note — This is purely my perspective as a practitioner with firsthand visibility into several working solutions in my career as a software consultant. Much of the vocabulary used in this series is of my own imagination and will surely cede to better nomenclature from the community. Moreover, many implementations I have seen in practice include multiple types of patterns discussed in this series.
Note — This commentary is neither definitive nor endorsed and is in no way representative of the views of my employer, Accenture.
While Terraform is primarily a configuration language that abstracts the APIs of another platform, there are logic expressions available in Terraform allowing developers to introduce inferences and abstractions, improving the flexibility (or general accessibility) of a solution, without introducing repetitive code or requiring deep technical knowledge on behalf of a module’s consumer. However, the tendency to see each of the world’s problems as a nail once one’s hammer is mastered can lead a team into a quagmire of complexity. On the other side, a draconian forbiddance of Terraform expressions can force teams into developing unreasonably large and repetitive codebase or splitting a team’s overall state into a vast wilderness of Terraform state trees.
In this post, we will discuss the most likely places for logic to appear, the major contributors to code complexity and developer confusion, and some gentle suggestions on how one might address or approach the spread of logic. We’ll wrap up with some general guidance and recommendations. It is worth reiterating that this is one person’s (severely non-exhaustive) perspective, and may be the most heretical or misguided post so far in the series. As such, I warmly welcome additional and dissenting viewpoints.
Common Contexts of Logic
Field Generation
This is some of the most common logic found in Terraform code. The most effective (and least problematic) field generation is string building via interpolation or map building over lists of input or output. The former is introduced to simplify the input fields of component blocks (i.e. named resources or modules), allowing for a greater number of fields to be constructed from fewer variables to be set by operators. This might be common for an organization that requires standardization around naming conventions and resource tagging[1] such as a repeated application or business codename that must appear as part of the standardized name of several managed resources, in their network and resource tags, or in the descriptions thereof.
Generating a map may be utilized to generate complex constructs from simpler module inputs and even combine them with the outputs of resources or remote modules, allowing an operator to keep input variables human-readable and the fields of modules or resources Terraform-readable.
Perhaps the most acceptable use of string interpolation is to clean up or directly manipulate the outputs of resources or 3rd-party remote modules as it is not always immediately possible to address otherwise. It is often cleaner to keep this logic in a locals block (see next subsection).
Local Variables
It seems most common to have one locals block per module. If all component blocks are in main.tf (and the locals block isn’t more than 15–20 lines[2]), these can be too. Otherwise, put in its own locals.tf file. It is strongly recommended to name your locals well; consider “self documenting code” where the names tell what relationship or purpose the generated variable addresses. As discussed above, the locals block is the most appropriate place for field generation including restructuring the outputs of resources or 3rd-party remote modules.
Externalized Logic
Terraform will most likely be embedded in your automation tool’s DSL, even if it amounts to yaml and freetext bash scripts. This unlocks the breadth of tools (or lack thereof) available to the execution context. In many popular platforms, these execution contexts are already containerized and even allow for custom container images to be leveraged for a job’s execution. This opens a world of possibilities to incorporate tools other than Terraform to perform operations on inputs and outputs. This may get messy if one leverages inline sed/awk commands or pipes, which brings me to the next point.
Consider using a friendlier scripting language like Python or Javascript if you have a lot of fancy logic to expand any inputs or analyze any outputs with code. These can be tested and stepped through with a debugging tool in ways Terraform cannot. This can even back a user interface or API, turning the system into fully self-service and event driven, perhaps negating the need for an automation tool at all.
Expressions and Functions
The for_each Expression
Definitely always use for_each with a map and with thoughtful, fully deterministic keys to ensure the state stays well identified and does not rely on a strict order. Ideally, these expressions are used sparingly and don’t show up in too many places in a given hierarchy[3].
Using a for_each over a consistently variable set of the same component block is certainly superior to trying to duplicate full component blocks, especially as you manage dozens or hundreds of components in this way. However, if you know you will only ever have a small (2–3), set number of resources (e.g. an active/passive setup for a particular tool to stay available during outages or help recover from disaster), it may not be worth the loss of clarity and extra syntax. Also, it can be destructive or cumbersome to move from one paradigm to another, as it does affect the full underlying state tree[4]. This is explored more below in the Conjectures & Conclusions section.
The count Expression
Pretty much never use count except as a way to conditionally include a block. The most common usage is to use a ternary statement inline within the component block.
The result will be either the presence of a list with a single element or nothing, so the output of a resource conditioned thusly requires the use of the list index notation. Referencing this output will also typically require the use of another ternary or the splat expression, leaking complexity into other parts of the codebase.
Moving from a count’ed component block to a for_each’ed component block (or one with neither) also suffers from the destructive/cumbersome attribute as before.
Regarding dynamic Blocks
These are also used along with a for_each or count statement. When resource fields are expressed as blocks, and when multiple blocks are allowed and required such that there is a high degree of variability in their use (examples may include disk attachments, network interfaces, or policy statements) and especially when the data to be passed to the block fields come from the outputs of resources or modules, it may be most prudent to leverage dynamic blocks. However, these should be used sparingly and never included in order to “future proof” (i.e. over-engineer) a module.
Furthermore, unless developing 3rd-party modules (or intending to be used as such), it may be best to avoid nesting dynamic blocks. This introduces a level of complexity most likely better served by parallel dynamic blocks, even if this introduces some (or significant) code repetition or other logic types (e.g. using count as a conditional, as described below).
The for/in(/if) Expression
Although this may be common inline with e.g. for_each expressions, it is perhaps cleanest to put them all in a locals block. It is also recommended to break into individual lines and not to nest for/in expressions. Create a named local variable for each expression, rather than putting it all in one compound expression. The result of each expression could then be used in outputs for better traceability when desired.
For non-primitive lists, consider restricting the usage of for/in purely to casting the list to a map with fully deterministic keys. This allows for a later for/in or for_each expression to act over a map, resulting in a more predictable and deterministic (and therefore more robust) behavior.
The if statement can improve code readability when iterating over a collection for different reasons and when items are conditionally considered (e.g. in the case of a list of virtual machines that optionally have an attached disk, where the module creates both instances and disks): rather than looping over the same list, create multiple lists with suggestive names that include the condition as an if statement.
Complex Variable Type Functions
These are used in variable definitions to define the module’s input variables that are not of type string, number, or bool (boolean). The most common non-primitive type constructors are list(), set(), map(), and object()[5]. Defining these is strongly preferred to leveraging the any type (or not including a type at all) but object definitions in particular can get complex, especially when including a default value.
The presence of these functions follow the structure of the module code that operates on the input variable: a count or for_each statement requires a list , set , or map type. Field values of both map and object types can be accessed using the bracket notation on the field key a la many common programming languages.
The length() Function
This can be used on strings or collection types. For the former, length() is common in conjunction with the substr() function, to ensure strings adhere to maximum length guidelines imposed by a particular platform or policy.
For the latter, length() may be appropriate in the context of providing more or richer details in output definitions. Introducing more complex logic based on length()[6] is perhaps as ill-advised, avoidable, and awkward as it seems to be rare.
String Matching Functions
The strcontains() function produces a boolean result based on the presence of a substring. This limited behavior results in limited benefit, and may be better served with an additional (if repetitive) input variable.
The related regex()/regexall() functions each return a list of matching substrings. These allow for richer matching, but are unlikely to be directly actionable, requiring additional expressions to make use of the outputs. This too may be better served with abstracting the logic as additional input variables or with externalized tooling.
The chomp() Function and the trim Functions
Ideally, the strings the code is operating upon is clean enough to use directly. The behavior of the chomp() and the various functions related to trim() in Terraform is cumbersome to test as a unit and very likely does not perform any better than other common tools (see section above on externalized logic). Then again, there’s something to be said for “defense in depth”, so even the use of external tools to clean up text input does not necessarily preclude the use of the trim functions[7].
Other String Manipulation Functions
It is most likely best to avoid complex string manipulation like nesting split(), replace(), join(), etc., except perhaps in cases in which one must manipulate output values to make proper use of them. Because Terraform functions are purely functional (rather than e.g. object oriented), chaining commands can make for less readable code (see additional points on functional behavior below). There is support for multi-line strings using a “heredoc” syntax, but this should be a rare if not overly contrived occurrence.
Directives
It appears the natural inclination is to avoid directives, especially in root modules or any modules that change often. A more common and perhaps accessible approach is to use a conditional expression or a for/in expression (see above), even though this may make for more characters in the code file.
There is the ability to introduce whitespace stripping with directives, which might not be well handled by another method. There may also be sound rationale to introduce directives or the “heredoc” syntax in the case of “workflow driven IaC automation” (in which some ticketing system passes in the parameters that were input into a web form by a user or other automated system). However, I would strongly recommend cleansing the textual variable inputs before one feeds them to Terraform with a more commonly used tool (as described in the section above on externalized logic, in this case e.g. sed/awk, Jinja, etc.).
Conjectures & Conclusions
Functional Programming
If you are familiar with functional languages like Lisp/Scheme/etc. or Haskell, it shouldn’t be too foreign of a concept to reason through function calls in Terraform. If you’re more familiar with the much more limited approach of many object oriented languages[8] to “lambda functions”, whereby pure functions are passed as arguments to specialized methods of collection types that iterate over the collection, sequentially calling said function on each iteration, the nesting behavior of inline function chaining in the functional programming paradigm may be both awkward and difficult to reason through. As such, it is strongly recommended that chained function calls be introduced as local variables (see the commentary above on Field Generation and Local Variables).
If your project is rife with function calls, I would recommend reading through a book on the subject, such as The Little Schemer (it’s worth the read even if your project has few functions). However, it may be most appropriate to reduce rather than endorse the use of functions.
Looping & Conditionals
It is best to fully consider the use of for_each versus multiple named (or even conditioned using count) modules/resources, especially with resource instances that require extra care in the event of any interruption, as changing from one paradigm to another is often destructive (i.e. the resource will be deleted from one location of the state tree and recreated elsewhere) or requires manipulation of the Terraform state object, directly, to align the state with the resulting shape expressed by the new code.
Conditionals (e.g. using the ternary operator as part of an expression) can be leveraged as a way to provide runtime validation around fields, ensuring the values set to the fields are appropriate types. Excessive use of this is mostly a traditional pattern, as earlier versions of Terraform didn’t include more appropriate validation patterns or a good way to control emptiness, e.g. the validation and the nullable fields of variable definitions, the try() and can() functions, or the optional modifier. Unfortunately, a significant number of fairly substantial entities haven’t fully progressed beyond these earlier versions (either literally or developmentally/culturally), and while this pattern is rather syntactically expensive and unsightly, it still appears regularly.
Complex Variable Types, A Slight Return
Leveraging different complex types at multiple places in a state hierarchy (rather than passing along the same constructs along it) may produce a kind of telescoping effect: the parent module of a child with an object type variable may be inclined to embed this object type as a subfield of its own object type variable. However, the natural opposite of this–passing along less complex objects–may have an equally loathsome result of complex logic all along the hierarchy to generate complex local variables that skirt any semblance of a type check and don’t exactly simplify the codebase.
Perhaps the best way to broker a balance between the two unfortunate extremes is keeping whatever modules consuming–i.e. decomposing into it’s fields and using them entirely (and most importantly not repackaging them into some other object to be passed further downstream)–the complex variables close to the active module, at least ensuring low repetition.
Next Time…
We’ll discuss style and convention in Terraform. As this is perhaps most likely to ignite an emotionally spirited debate (see Tabs v. Spaces), I will attempt to generally revert back to observation and avoid over-editorializing.
Footnotes
[1] For cost attribution and/or as the subject of a network configuration.
[2] Your mileage may vary.
[3] Quantifying “too many” left as an exercise for the reader.
[4] Not the somewhat logical one we’ve considered up to now that hadn’t really considered the difference a for_each or count statement makes to a component.
[5] There is also tuple(), but the type tuple seems to be identical to the type list.
[6] — A poorly contrived example of which may include triggering a quota increase request when a certain threshold of inputs or outputs is reached.
[7] Perhaps the point is other tools may be more reliable than Terraform for cleaning up strings if only one tool is to be relied upon.
[8] At least c.a. the last time I professionally developed such software.
