Deploy Terraform & Serverless Framework in Perfect Harmony

On our team, we recently switched to a fully automated deployment process where both the infrastructure and applications are deployed through our CI/CD system. There are so many benefits to doing this — reproducibility, consistency, audit trails, change management. And the list goes on. If you’re not doing it, I highly recommend spending the time to get there.

It’s not an easy road, but here is how we got there using both Terraform and Serverless Framework.

Originally we like many teams were writing code and deploying it manually running serverless deploy locally. And the same for our infrastructure — terraform apply with local state checked into git. We would then take the generated output from Terraform, like IAM policies or other resources ARNs, and manually add those to the serverless.yml file.

Both of processes worked but did not scale when your team grows to more than a few people.

At first, we moved the Serverless deployment to our CI/CD pipeline as we were already using it to deploy Docker containers through ECS. So it was relatively easy to just add that the deployment pipeline.

But we were still deploying infrastructure locally. This process was a little different because the environment that deploys the infrastructure needs quite a bit more permissions than the environment that deploys the application. In fact a LOT more permissions.

The first step was to figure out what permissions the infrastructure environment needed and give those environments the correct assumed IAM roles.

The second step was getting Terraform and Serverless to work in harmony. What I mean by that is we wanted the resources created inTerraform to be available to Serverless.

So if I create a function IAM policy in Terraform, I want to use that within the serverless.yml file automatically. Or if I create a DynamoDB table in Terraform, I want my serverless application to know and use it.

To do all of these we need to capture the output of the Terraform deploy and save it to a JSON file which can be used both in the serverless.yml and application itself.

Here’s how we do this:

1. We first need to define what values Terraform will output.

// _output.tf
output "dynamodb_table_name" {
  value = "${aws_dynamodb_table.my_table.name}"
}
output “role_arn" {
  value = "${aws_iam_role.app.arn}"
}

2. Next, we need to apply Terraform and send the output into JSON format to “output.json”. We also use jq to format the output into key/value pairs for easy consumption.

terraform apply -auto-approve
terraform output -json | jq 'with_entries(.value |= .value)’ > config.json
// would output
{
  "dynamodb_table_name": "my_table",
  "role_arn”: “arn:..."
}

3. We can now use this “config.json” file in both the serverless.yml require it into our application.

// serverless.yml
provider:
  name: aws
  runtime: nodejs8.10
  role: ${file(config.json):role_arn}
// NodeJS application
let config = require('./config.json')
let params = {
  TableName: config.dynamodb_table_name,
  Item: {
    userId,
    name
  }
}
...

We use this pattern for all resources created with Terraform where the application needs to know about the resources. We’ve found it a really powerful pattern that works very well across all our environments.