Lessons learned when upgrading to Terraform 0.12- part 2

In a previous post, I went through a list of things to do when upgrading to Terraform 0.12. These consisted of:

1. fixing breaking changes
2. using first-class expressions
3. using improved conditionals
4. Using dynamic blocks to reduce code repetition
5. Upgrading self-contained modules

In this post, I’ll mainly concentrate on using rich types and type constraints.

Primitive types

A primitive type is, as the definition says, a simple type that is not made from any other types. Primitive types are string, number and bool.

Terraform automatically converts numbers and boolean values to string and vice-versa when needed. It’s good practice, however, to explicitly define your variable type to avoid ambiguity.

Complex types

A complex type is a type that groups multiple values into a single value.

There are 2 complex types:

1. collections: list, map, set
2. structural: object, tuple

Collection types are usually multiple values of the same type grouped under a single collection e.g.

// list of integers
gitlab_lb_ports = [22, 80, 443]
// map of [string,integers]
variable "availability_domains" {
  description = "ADs where to provision non-OKE resources"
  type        = "map"
  default = {
    bastion = 1
  }
}

Occasionally, you may also need to mix types. You can either create custom structural types or use tuples. We’ll come back to structural types shortly but let’s see what tuples offer us.

A tuple allows you to define a sequence of elements and their respective types e.g.

//vcn_name, cidr, create_nat_gateway         
variable vcn = tuple([string,string,bool])

When using tuples, your values must match the same number and type of parameters in the specified order e.g.

// bad tuple based on the above definition
vcn = [false,"my_vcn","10.0.0.0/16"]
//good tuple
vcn = ["my_vcn","10.0.0.0/16",false]

As you can see, the order and type of variables matter with tuples.

A structural type allows you to create any object schema e.g.

variable "oke_identity" {
  type = object({
    compartment_id = string
    user_id        = string
  })
}
variable "oke_bastion" {
  type = object({
    bastion_public_ip         = string
    create_bastion            = bool
    enable_instance_principal = bool
  })
}

Additionally, a structural type can also contain an attribute of type collection e.g.

variable "oke_general" {
  type = object({
    ad_names     = list(string)
    label_prefix = string
    region       = string
  })
}
variable "oke_network_vcn" {
  type = object({
    ig_route_id                = string
    is_service_gateway_enabled = bool
    nat_route_id               = string
    newbits                    = map(number)
    subnets                    = map(number)
    vcn_cidr                   = string
    vcn_id                     = string
  })
}

When an attribute consists of a collection type, you must also specify the type of values as highlighted above.

Occasionally, you may need to also pass a collection of mixed types which you can then retrieve via the collection functions. In this case, you can use the ‘any’ type e.g.

variable "node_pools" {
  type        = map(any)
  description = "number of node pools"
}

allows us to pass values like these:

node_pools = {
  "np1" = ["VM.Standard.E2.1", 3]
  "np2" = ["VM.Standard2.1", 4]
  "np3" = ["VM.Standard.E2.2", 5]
}

Now, that you know how to define the types, 3 final questions remain:

1. how do you create structural types
2. how to use/reuse them when using modules/sub-modules
3. how to specify root variables

Creating structural types

If you are familiar with Object-Oriented (OOP) principles, you can define your types accordingly. e.g. in the terraform-oci-oke project, we define a set of objects that group the variables:

variable "oci_base_identity" {
  type = object({
    api_fingerprint      = string
    api_private_key_path = string
    compartment_id       = string
    compartment_name     = string
    tenancy_id           = string
    user_id              = string
  })
}
variable "oci_base_ssh_keys" {
  type = object({
    ssh_private_key_path = string
    ssh_public_key_path  = string
  })
}

You would usually group attributes of a particular object/resource or related parameters together e.g. oci_base_identity contains the identity-related attributes whereas oci_base_ssh_keys contains the public and private ssh key pair paths.

Using structural types with modules/submodules

If you have several modules, I found it a good practice to prefix the variable name with the module name so I know where these variables are used.

//base module variables
oci_base_identity = {
...
}
oci_base_ssh_keys = {
...
}
oci_base_general = {
...
}
//oke_module variables
oke_general = {
...
}
oke_bastion = {
...
}
//oke_network variables
oke_network_vcn = {
...
}

Specifying root variables

This one is probably going to be a bit contentious but here’s my take:

1. if your terraform project is a reusable module and will be reused by another project, use structural types in your root variables
2. if your terraform project will be used to actually create infrastructure and not intended for reuse, use a flat structure and simple/collection types when possible

In the 1st case, developers will be writing code against your root variables (effectively your module’s APIs). As such, they’ll be expected to understand your variables and their types in order to effectively call and reuse your module. They might even read your code.

Using structural types also helps in reducing the amount of boilerplate code they would have to write e.g. in Terraform 0.11, without object types, we had to do this to define and reuse the base module:

module "base" {  
  source = "./modules/base"   
  
  # identity   
  api_fingerprint      = "${var.api_fingerprint}"
  api_private_key_path = "${var.api_private_key_path}"
  compartment_name     = "${var.compartment_name}"  
  compartment_ocid     = "${var.compartment_ocid}"  
  tenancy_ocid         = "${var.tenancy_ocid}"  
  user_ocid            = "${var.user_ocid}"  
  ssh_private_key_path = "${var.ssh_private_key_path}"
  ssh_public_key_path  = "${var.ssh_public_key_path}"   
  # general  
  label_prefix = "${var.label_prefix}"  
  region       = "${var.region}"   
  # networking  
  newbits                = "${var.newbits}"  
  subnets                = "${var.subnets}"  
  vcn_cidr               = "${var.vcn_cidr}"  
  vcn_dns_name           = "${var.vcn_dns_name}"  
  vcn_name               = "${var.vcn_name}"  
  create_nat_gateway     = "${var.create_nat_gateway}"
  nat_gateway_name       = "${var.nat_gateway_name}"  
  create_service_gateway = "${var.create_service_gateway}"  
  service_gateway_name   = "${var.service_gateway_name}"   
  # bastion  
  bastion_shape               = "${var.bastion_shape}"   
  create_bastion              = "${var.create_bastion}"  
  enable_instance_principal   ="${var.enable_instance_principal}"  
  image_ocid                  = "${var.image_ocid}"  
  image_operating_system      = "${var.image_operating_system}"   
  image_operating_system_version = "${var.image_operating_system_version}"   
# availability_domains
availability_domains = "${var.availability_domains}"}

Compare this in Terraform 0.12 when using object types:

module "base" {  
  source = "./modules/base"   
  
  # identity  
  oci_base_identity = local.oci_base_identity
  # ssh keys  
  oci_base_ssh_keys = local.oci_base_ssh_keys   
  # general oci parameters  
  oci_base_general = local.oci_base_general   
  # vcn parameters  
  oci_base_vcn = local.oci_base_vcn   
  # bastion parameters  
  oci_base_bastion = local.oci_base_bastion
}

and you can do the dirty work in locals:

locals {   
  oci_base_identity = {    
    api_fingerprint      = var.api_fingerprint    
    api_private_key_path = var.api_private_key_path      
    compartment_name     = var.compartment_name    
    compartment_id       = var.compartment_id    
    tenancy_id           = var.tenancy_id    
    user_id              = var.user_id  
  }
   
  oci_base_ssh_keys = {    
    ssh_private_key_path = var.ssh_private_key_path       
    ssh_public_key_path  = var.ssh_public_key_path  
  }
 ....
}

Finally, as you add new features to your module and publish new versions to the registry, you can use Semantic Versioning to ensure that users of your modules and their downstream users do not run into compatibility problems. This is the approach we took in the reusable terraform-oci-base module. Reusing it will then look like the following:

module "base" {
  source  = "mybase"
  version = "1.0.1"
}

In the 2nd case, the end users are not developers and they’ll need to create variable files when running Terraform. It is therefore easier for them to create variable files if the structure is flat. My inspiration for this approach is TOML, which “aims to be minimal configuration file format that’s easy to read due to obvious semantics.”

While the terraform variable file structure is not strictly TOML-compliant, the purpose here is to make it easy for users to create a configuration file, so keeping a flat structure for your root variables, in this case, is preferable. This is the approach we took in the terraform-oci-oke module.

Terraform tip: summing a list of numbers using for

I found this little gem while trying to figure out how to sum a list of numbers.

Summary

Use the following approaches when using rich types:

1. If your project is a reusable module, use rich types in your root variables. Otherwise, use a flat structure to keep it simple to use for end-users.
2. Explicitly specify the type of your variables
3. Group your variables using OOP principles
4. Prefix your variable names with the module name where they are used
5. Use collections when possible and specify the type of values they will be using
6. Nice Robot helpfully suggested doing concatenation in locals. Take it further and do your object initialization in locals.