Sharing global stack data with Terraform

Photo by James Coleman on Unsplash

So let’s say you have decomposed your Terraform infrastructure into small individual and mostly independent stacks. What is the best method to share common data between them? In this article we will discuss a way.

For this demonstration we will assume there are two services in separate stacks; service-a and service-b. Both require an ALB on same the subnets in the same VPC.

module service_a_alb {
  source = "terraform-aws-modules/alb/aws"

  name = "service-a"

  load_balancer_type = "application"

  vpc_id          = "vpc-12345"
  subnets         = [
    "subnet-foo",
    "subnet-bar",
    "subnet-baz"
  ]
  ...
}

module service_b_alb {
  source = "terraform-aws-modules/alb/aws"

  name = "service-b"

  load_balancer_type = "application"

  vpc_id          = "vpc-12345"
  subnets         = [
    "subnet-foo",
    "subnet-bar",
    "subnet-baz"
  ]
  ...
}

You can see the vpc_id and subnets are duplicated and shared between the stacks. The obvious thing to do now would be to use data resources:

data aws_vpc vpc {
  default = false
  filter {
    name   = "tag:Name"
    values = ["MY-VPC"]
  }
}

data aws_subnet_ids public {
  vpc_id = data.aws_vpc.vpc.id
  filter {
    name = "tag:Name"
    values = [
      "PUBLIC-*",
    ]
  }
}
module service_a_alb {
  source = "terraform-aws-modules/alb/aws"

  name = "service-a"
  load_balancer_type = "application"

  vpc_id          = data.aws_vpc.vpc_id
  subnets         = data.aws_subnet_ids.public.ids
}

data aws_vpc vpc {
  default = false
  filter {
    name   = "tag:Name"
    values = ["MY-VPC"]
  }
}

data aws_subnet_ids public {
  vpc_id = data.aws_vpc.vpc.id
  filter {
    name = "tag:Name"
    values = [
      "PUBLIC-*",
    ]
  }
}
module service_b_alb {
  source = "terraform-aws-modules/alb/aws"

  name = "service-b"

  load_balancer_type = "application"

  vpc_id          = data.aws_vpc.vpc_id
  subnets         = data.aws_subnet_ids.public.ids
}

but now the data resources are duplicated and AWS has just deprecated one of your availability zones. Resources can no longer be provisioned in it. The last thing you want to do is to go into each stack and edit the data resources. What to do?

Move the data resources into their own stack:

data aws_vpc vpc {
  default = false
  filter {
    name   = "tag:Name"
    values = ["MY-VPC"]
  }
}

data aws_subnet_ids public {
  vpc_id = data.aws_vpc.vpc.id
  filter {
    name = "tag:Name"
    values = [
      "PUBLIC-*",
    ]
  }
}
output vpc_id {
  value = data.aws_vpc.vpc.id
}
output public_subnet_ids {
  value = data.aws_subnet_ids.public.ids
}
terraform {
  backend "s3" {
    bucket  = "terraform.example.com"
    key     = "{[ env_name ]}/{[ aws_region ]}/{[ vpc_name ]}"
    region  = {[ aws_region ]}
    encrypt = true
  }
}

data terraform_remote_state env {
  backend = "s3"
  config = {
    bucket  = "terraform.example.com"
    encrypt = true
    key     = "${var.env_name}/${var.aws_region}/${var.vpc_name}"
    region  = var.aws_region
  }
}
locals {
  public_subnet_ids = data.terraform_remote_state.env.outputs.public_subnet_ids
  vpc_id = data.terraform_remote_state.env.outputs.vpc_id
}
module service_a_alb {
  source = "terraform-aws-modules/alb/aws"

  name = "service-a"

  load_balancer_type = "application"

  vpc_id          = local.vpc_id
  subnets         = local.public_subnet_ids
}

data terraform_remote_state env {
  backend = "s3"
  config = {
    bucket  = "terraform.example.com"
    encrypt = true
    key     = "${var.env_name}/${var.aws_region}/${var.vpc_name}"
    region  = var.aws_region
  }
}
locals {
  public_subnet_ids = data.terraform_remote_state.env.outputs.public_subnet_ids
  vpc_id = data.terraform_remote_state.env.outputs.vpc_id
}
module service_b_alb {
  source = "terraform-aws-modules/alb/aws"

  name = "service-b"

  load_balancer_type = "application"

  vpc_id          = local.vpc_id
  subnets         = local.public_subnet_ids
}

By now you might be thinking well the references to the data stack are duplicated. Yes they are but let’s go back to the issue of the deprecated az. To filter out the bad subnets you only need to make the change in one place:

locals {
  excluded_public_subnets = [
    "subnet-baz"
  ]
}
output public_subnet_ids {
  value = [
    for subnet_id in data.aws_subnet_ids.public.ids:
    subnet_id if ! contains(local.excluded_public_subnets, subnet_id)
  ]
}

the stacks for service-a and service-b do not have to change.

Please let me know your thoughts and opinions on this. How do you manage shared data in a decomposed Terraform environment?