Photo by Sean Benesh on Unsplash

Automated Terraform configuration imports from AWS

So you didn’t start day zero with your infrastructure as code? No problem. Our team at Koan didn’t either. But keeping infrastructure in version control comes with serious benefits, and if you’re already using AWS, the journey towards managed infrastructure may be easier than you think.

In a previous post we shared how we used Terraform and Ansible to start moving infrastructure into version control. Our initial work was manual, repetitive, and ripe for automation — music to a development team’s ears.

Here’s how it went down.

Motivations

Infrastructure as code is a good thing. Instead of infrastructure changes only turning up in audit logs after a well-meaning operators has already applied them (through the AWS dashboard, say), tools like terraform let a dev team propose, review, and adjust changes before they go live.

Software is tested and reviewed before being deployed, and infrastructure should be the same way.

Much of Koan’s early infrastructure was provisioned manually through the AWS dashboard. It got the job done, but it also made it difficult to track changes or understand our resource usage as a whole. With Terraform in the mix, the rules change:

1. Avoid the AWS dashboard as much as possible
2. Use terraform and the aws command-line interface instead

The only catch is that the infrastructure needs to be in Terraform before it’s ready to be used. Terraform is happy to import state, but generating resource configurations is a bit more of a project.

Organizing imported resources

When we first set out to import our infrastructure, we spent a fair amount of time considering how our configuration should be organized.

Based on our experience using Terraform in other shops and environments, our team’s inclination was to group Terraform-managed resources around the applications and services that use them. This keeps usage and dependencies clear, and it’s where we want to get to.

With an existing environment already on the books, however, we settled on the intermediate step of grouping imported resources by the corresponding AWS service. This allowed us to focus our import efforts on a single service at a time, letting us handle all the research and scripting needed to, say, import all of our DNS records from Route 53 before moving on to the next resource type.

The initial configuration wasn’t pretty, but shuffling configuration and state around in Terraform is a relatively straightforward process. We left it for another day.

Importing resources into Terraform

Next, we worked out the actual import. Terraform’s built-in import command works well for attaching existing resources to the Terraform state, but it doesn’t yet generate the corresponding configuration (this seems likely to change in a future release). Fortunately, a few shell scripts and a little bit of elbow grease can quickly make up the difference.

While the details vary between resources and services, the general approach is to:

1. Create a stub definition in the Terraform configuration
2. Import the corresponding resource
3. Replace the stub with the imported state

The AWS services used at Koan follow mostly-standard API conventions. “Mostly” doesn’t quite cut it in shell scripts, however, and as we automated the import process we found ourselves tailoring common boilerplate to the nuances of the service in question.

This usually began by completing one or two resources from a given service by hand — manual, but getting the ID parameters and state format right before running a large batch of imports saved considerable cleanup and rework.

Automating AWS resource imports

Our import boilerplate uses a variable called $tf_resource_type to assign the imported AWS resource to the correct Terraform resource type (for a Route 53 record, for instance, $tf_resource_type would be 'aws_route53_record') and a $resource_name parameter to hold the dynamically-generated name that will represent the resource in our Terraform config.

Resource naming varies between services but typically involves slugifying the name or ID of the corresponding AWS resource. In the Route 53 example, the $resource_name of the A record for our website at koan.co) might be mapped to koan-co-a.

Using these variables, translating the steps to import an AWS resource into the outlines of a shell script loks something like this:

# write a stub definition to a terraform configuration file
$ echo "resource $tf_resource_type \\"$resource_name\\" {}" > aws_service.tf
# import the corresponding resource
$ terraform import $tf_resource_type.$resource_name
# append state definition to the configuration file
$ terraform state show -no-color $resource_name >> aws_service.tf
# delete the original stub
$ sed 1d aws_service.tf
# and manually edit the imported resource until `terraform plan` comes up
# error-free and clean
$ vim aws_service.tf # emacs is cool, too.

Converting Terraform state into configuration

New state imported via terraform import and printed out via terraform state show isn’t quite ready to be used as a Terraform configuration. As our import progressed, we frequently needed to apply several different cleanup steps:

1. Stripping identifiers like ids and arns. Terraform configuration and state aren't quite one-to-one, and for good reason. Separating the two allows the configuration to declare resources without worrying about how they map to "live" resources.
2. Rewriting mismatched Heredoc delimiters. At time of writing, terraform state show prints heredoc strings with a <<~EOT tag that’s one ~ away from the closing EOT.
3. Fixing string-escaping in HCL keys. We ran into several cases where state show output included un-quoted fields (iam:AWSServiceName instead of "iam:AWSServiceName") that terraform wouldn't parse.

When we only had a few resources defined in a service, we tended to perform these tasks by hand. In higher-volume imports, however, we customized the import scripts to clean up the output from terraform state show directly:

$ terraform state show -no-color "aws_kms_key.${name}" \\
  | sed 's/kms:GrantIsForAWSResource/\\"kms:GrantIsForAWSResource\\"/' \\
  | sed 's/kms:CallerAccount/\\"kms:CallerAccount\\"/' \\
  | sed 's/kms:ViaService/\\"kms:ViaService\\"/'

In general, we found some combination of jq, sed, and vim macros totally sufficient to translate the (JSON) state back into valid Terraform HCL. And the rest of the time, we weren’t shy about rolling up our sleeves editing things manually.

Scripting imports

Once we were fairly confident in the steps needed to import resources from a specific AWS service, we wrapped the import process in discovery logic covering all resources provided by the service. The details varied a bit from one service to the next, but the process was fairly consistent:

1. Discover AWS resources of a single $tf_resource_type
2. Map AWS resources to Terraform-compatible $resource_name parameters
3. Iterate through each $resource_name and apply the (now-automated) import process

1. Discover resources from the AWS CLI

In this stage, we typically used aws-cli to list resources and jq to prep them for consumption. For instance, enumerating IAM Groups for import might look something like this:

$ aws iam list-groups \\
  | jq -r '.Groups[] | (.GroupName | ascii_downcase) + " " + .Arn'
admins arn:aws:iam::123456789:group/Admins
analysts arn:aws:iam::123456789:group/Analysts
operators arn:aws:iam::123456789:group/Operators

2. Map AWS resources to Terraform-friendly names

AWS resource identifiers vary from one service to the next, and it’s not always possible (or desirable) drop them straight into Terraform configuration.

While some resources (our IAM groups) have human-friendly names, others are identified by opaque alphanumeric IDs.

We considered using these directly but backed away fast — legibility matters, and opaque IDs tell a human reader nothing about what they represent.

In these instances we typically drew up hand-coded maps between AWS identifiers and the more-descriptive names that would identify them in Terraform. For example:

# Distribution IDs and terraform resource names to import
DISTRIBUTIONS='
EABC3I6A7KJF5Q static-koan-co
EABC0VYMHHMHP app-koan-co
EABCMOYKPFC5X stage-koan-co
...
'

Instead of importing resources directly from the aws CLI, scripts with manual mappings iterated over the map instead.

3. Apply the (automated) import

With resource discovery and mapping in place, imports came down to scripting up the same process we’d previously run by hand.

Iterate over the resources in AWS and:

1. Create a stub configuration in Terraform
2. Run terraform import
3. Replace the stub with the imported state

Then, if terraform plan came up empty and error free, we ran terraform fmt against the results and checked them into git.

# Make sure there are no changes (or that any proposed
# changes are understood and acceptable).
$ terraform plan
# Or limit to newly-imported resources
$ terraform plan \\
  -target=aws_route53_record.koan-co-a \\
  -target=aws_route53_record.koan-co-a

Importing resources across regions

One “gotcha” we encountered came when we imported resources distributed across different AWS regions. Terraform provides several ways to address this, including:

1. Defining additional aws providers for additional regions
2. OR consolidating regions in their own modules
3. OR maintaining separate configurations for each region

Modular organization assigned per-region and per-environment feel like a taxonomically-satisfying future state, but with the focus on documenting existing resources we left them for another day. That brought us back to the first option: a single, monolithic configuration with multiple providers defined.

In general, imports from outside our default region needed two additional steps:

• Namespacing resources that are present in multiple regions with the same human-friendly resource name (e.g. aws_acm_cert.my-cert-us-east-1)
• Using terraform import -region= to override the region on import, or specifying the appropriate provider when generating a stub block: resource aws_acm_cert.my-cert-us-east-1 { provider = aws.us-east-1 }

Per-region organization won’t be this way forever. But the combination of additional providers and namespaced resources kept us moving forward towards our goal. Organization could come after.

Next steps

Once Terraform contained a working image of our AWS infrastructure, we were on to the hardest part of all: cleaning up.

Some of the high-priority projects identified during import include:

• Cleaning house. The documentation process turned up a variety of legacy resources that we no longer needed. These ranged from mostly harmless (hand-rolled S3 buckets with minimal data usage) to outright dangerous (hand-rolled IAM policies granting access to who knows what), and they all needed to go.
• Parameterizing resources. Our imported resources reference external identifiers (IDs, ARNs, hostnames, etc) rather than each other. It’s better than nothing, strictly speaking, but it leaves resources referencing one another through strings rather than human-friendly references. These we fixed manually as the configuration evolved.
• Organization. Importing resources per service is an efficient way to document them, but it doesn’t say much about the mapping between resources and the systems in our domain. Once we cleaned house, we could finally begin collecting the remaining parts into workspaces and modules mapped to the services we actually run.

And this was just the start. While we’d documented the most crucial elements in our infrastructure, the journey to evolve and scale it has only just begin.

Ready to take on technical challenges like these while helping us change change how teams deliver their goals? Koan is hiring, and we’d love to hear from you!