Combining Terraform and the Serverless Framework Gracefully

DISCLAIMER: I am not an expert in Terraform, but after scraping the web and hacking my own solutions, this was the best I could come up with. If you are an expert and want to suggest something to improve the code shown on this post, please be my guest.

BEFORE YOU CONTINUE, I suggest you read Making Terraform and Serverless framework work together and Deploy Terraform & Serverless Framework in Perfect Harmony posts. Both are very nice and helped me climb my way up on the Terraform + Serverless combination. My post aims to extract the best of the two articles and merging them into one.

In this post, I will walk you through how to use both Terraform and the Serverless Framework to create independent parts of your Cloud stack. For the sake of this example, I will use AWS as my Cloud Provider.

If you want to follow along, you will need to set up these things first:

If you don’t want to follow along, just grab your popcorn and enjoy your read!

Although most people think this is a sort of oil and water relationship, I dare to say this is more of a beer and snacks relationship. It’s impossible not to love this combo if you get it right. Even the guys who build the Serverless Framework advise that Terraform should be used in conjunction with the Serverless Framework, but not many people — at least from what I have seen — follow this advice.

It all started when someone asked me what the dividing line for creating resources with either Terraform or the Serverless Framework was. This is what I am going to try to answer on this post.

Initially, I used to think we should use one tool only if the project is purely Serverless. Managing one tool is easier for many reasons, including less complexity on the CI/CD pipeline, only one tool to learn, etc. But what happens when it is way too complicated/verbose to create one resource using this one tool? Also, if creating a given resource X is way easier on tool B, why should we pull our hairs out to create resource X on tool A?

And that’s how I started thinking the Serverless Framework is not the one-size-fits-all type of thing: don’t get me wrong though. I simply love the Serverless Framework, but I think that creating resources on it is simply too verbose since it heavily relies on CloudFormation. And we want to avoid writing CloudFormation templates as much as possible, don’t we? That’s actually the reason these frameworks exist and this is very likely the reason you ended up here. We should only write CloudFormation templates if we have to.

If you need to create a DynamoDB table in the Serverless Framework, here’s what you’d need to add on your serverless.yml file (or create a separate file and reference it on the resources section, but you’d still need to write CloudFormation manually regardless):

I mean, seriously? 15 lines to create a very minimal table? What if your function needs to create more tables? What if you need to create SQS Queues, SNS Topics, IAM Roles? Or what if multiple functions need to access the same resources, which function should be responsible for creating them? Argh! Enough. Things can get messy and unmaintainable very fast.

Since the Serverless Framework is very powerful, it has the ability to reference values from other CloudFormation stacks, meaning these stacks can be created independently, either manually or by some other tool, keeping the serverless.yml file clean and clear.

Using Terraform to create Infrastructure (as Code)

After reflecting for a little while on where the dividing line lived, I decided to use Terraform to create all kind of resources but Lambda functions and AWS API Gateway APIs, I went ahead and created a proof of concept to see how feasible it was to put Terraform in the game. Surprisingly enough, this went smoother than I expected.

In Terraform, in order to create the same table as mentioned above, we need only 10 lines of code:

This is a 33% reduction already. Of course, we don’t see much difference for small numbers, but try to see the bigger picture.

If we need to create a custom IAM role, it would look something like this in the Serverless Framework:

The snippet above was copied and pasted from the official docs in case you want to check it out for yourself.

In Terraform, similar (I said similar because these two snippets don’t do exactly the same things, but for the sake of this post I think you get the idea) behaviour can be easily achieved as:

You now must be asking yourself where that “lambda_assume_role.json” comes from. That’s a simple snippet, and, unfortunately, this needs to be written in CloudFormation (either in JSON or YAML). Yes, I know, not everything is perfect. Here’s what the template looks like:

You may be able to get the idea by now: writing Terraform code, generally speaking, is way shorter than writing Serverless Framework.

Time to Combine them!

Now that you see the main differences between the two tools, it’s time to combine the best of each and create our stack.

The real drawback that I see is that in order to create a CloudFormation stack we need at least one resource, which forces us to write CloudFormation templates once again. Once this is done though, we can use the same CloudFormation stack to output values that the Serverless Framework can make use of, even if the resources themselves are not managed by the CloudFormation stack!

Since I needed to create an SQS Queue on the project I was working at, I decided to keep it for this demo.

Do you see the Outputs section on the CloudFormation template? That is very important! Since this is a file managed by Terraform, it means we can inject pretty much anything we want and use those values in the Outputs section in our serverless.yml file seamlessly!

In the Terraform CloudFormation stack, we named our stack todo_list_outputs. All we need to do now is reference this stack on the serverless.yml file and use dot notation to access the resource we are after:

If you’ve downloaded the code for the Terraform project, you can navigate into it and run terraform init && terraform apply -auto-approve to see the magic happen. The outputs section of the CloudFormation stack will look something like this:

If you now navigate to the Serverless Project, just run sls deploy. Once it’s finished, you should be able to see a new API under API Gateway, a table in DynamoDB called dev-todo-list.

If you want to run a test, make a POST request against the endpoint available in API Gateway with a body containing at least a key named “id”. Example request:

curl -X POST -H ‘content-type:application/json’ -d ‘{“id”: “123”, “task”: “walk the dog”, “done”: false}’

You should see the item added in DynamoDB, proving that both Terraform and the Serverless framework are working properly and that all the resources have been created successfully.

Congratulations if you’ve made it to this line! You are now capable of managing your resources with both Terraform and the Serverless Framework!

If you feel like cleaning things up, just run terraform destroy -auto-approve on the Terraform directory and sls remove on the Serverless directory.

By adopting this approach, you completely decouple the infrastructure from your Lambda functions, meaning N functions can consume the outputs of a single Terraform project (i.e many functions publishing to the same SQS queue, writing to the same DynamoDB table, etc.).

And now, finally, let me get to answer the question. My general rule of thumb is:
Create all the infrastructure related things with Terraform (Queues, Topics, Databases, Tables, VPCs, IAM Roles, etc.)
Create functions and hook up events (like API Gateway, s3, DynamoDB Streams, SQS, SNS and so on) using the Serverless Framework

I hope you’ve enjoyed it and I look forward to reading your thoughts on it. Constructive criticism, compliments and feedback are always more than welcome.