Time-travelling with Terraform

Photo by Roger Ce on Unsplash

Recently I had to unwrap some hairy old Terraform code which was difficult to reason with and extremely hard to cleanly adjust in situ. The issue was that a module contained its own provider configuration, and the reason for that is that the resources the provider was manipulating were created by that module itself. This had accreted in place and wouldn’t work if deployed from scratch. The module did two things:

1. Create a database server and some database users
2. Grant roles to users on some databases

The problem is that the GCP API for Cloud SQL does not extend to SQL operations. Hence, while we could use the Google Terraform provider to provision the DBMS, an alternative solution was needed for rights management. Doing this with null providers and remote-exec provisioners is a traditional approach, but this is fraught with difficulty and is not recommended for many reasons (one of which is to ensure that said grants can be removed again using the usual Terraform lifecycle becomes very painful). As such we used a dedicated MySQL provider, in this case, petoju/mysql. This provider, however, needs to be configured with the database endpoint, which isn’t known before the server is built.

What this looked like at the time was something like:

calling module:

module "database" {
  source = "./modules/database"
  project = XXX
  region = XXX
  # we don't configure the mysql provider here at all
}

submodule:

terraform {
  required_version = "1.3.6"
  required_providers {
    mysql = {
      source  = "petoju/mysql"
      version = ">= 3.0.48"
    }
  }
}

google_sql_server_instance "my_database" {
  name = XXX
}

# we need to give the provider something before the DBMS is built
data google_sql_server_instance "my_database" {
  name = XXX
}

resource "google_sql_user" "user" {
  instance = data.google_sql_server_instance.my_database.id
}

resource "mysql_grant" "grant" {
  user = google_sql_user.user.id
}

In-module providers have long been discouraged, as these are treated as “legacy” modules and denied access to such niceties as for_each and depends_on.

How do you step back in time to let the calling module know details about things it hasn’t built yet?

The answer, it turns out, is that Terraform is now clever enough to plan ahead. So long as all of the resources that your provider is going to depend on are built using the module you’re calling, it is possible to expose their future values as outputs and Terraform is now clever enough to defer evaluation of them until they exist.

First, we yank the module provider out into the root module. Now, instead of using a data resource to represent the database, we can use module outputs:

output "endpoint" {
  value = "${google_sql_database_instance.my_database.private_ip}:3306"
}
output "user" {
  value = google_sql_database_instance.my_database.user
}
output "endpoint" {
  value = google_sql_database_instance.my_database.password
}

And then in the root module, your provider config is now as simple as:

terraform {
  required_version = "1.3.6"
  required_providers {
    mysql = {
      source  = "petoju/mysql"
      version = ">= 3.0.48"
    }
  }
}

provider "mysql" {
  endpoint = module.database.endpoint
  user = module.database.user
  password = module.database.password
}

Now, even though the database doesn’t exist at plan time, Terraform considers it sufficiently well-described that the MySQL provider can configure itself once the DBMS has been built. And the module gets its provider configuration passed into it rather than computing it itself, so it’s a first-class citizen again and the likes of for_each work against it.

Here’s what the new flow looks like:

Terraform’s documentation on this issue appears to be out of date: work to improve the way that Terraform constructs its plan graph landed in 2022, so it has been possible for some time for providers to depend on objects that don’t yet exist. This is nice, as it made for a much cleaner and more robust module in my case with relatively minimal changes.