Terraforming Our API Gateway

At Apartment List, we believe we can create the best experience possible to help every renter find a home they love. The Platform team’s mission is to enable other engineers to focus on features that delight by creating an enjoyable development experience at the company. We’ve learned a lot of stuff while rearchitecting our API to scale for growth, and adopting Terraform was a big part of that. We wrote some words to share!

About a year ago, we set out to build an API Gateway at Apartment List using AWS’s API Gateway offering. With most of our services hosted through Heroku, our own AWS footprint with applications serving production traffic was decidedly low, so this was, in a way, also a foray into learning to get good at managing things in AWS.

In an effort to learn each component of the API Gateway, we relied heavily on using the console for creating many of our resources. For each endpoint, that involved at minimum (1) creating the resource path, (2) adding a resource method, and (3) setting the integration request. This turns out to be a repetitive and arduous task when you have many endpoints that go out to different services. This situation wasn’t ideal for a couple of other reasons:

1. The API gateway and its resources weren’t easily reconstructible.
   What if some series of unfortunate events (an errant command here, a button click there) deleted the gateway altogether? It’d take forever to rebuild all the connecting pieces, and even then it’d be difficult to guarantee how faithfully we can rebuild it. Oy vey.
2. Creating resources via the console often had side effects that were difficult to track.
   For instance, when setting an integration request to a lambda function, the console will helpfully try to add the gateway resource to the lambda function policy.
3. There was no changelog of resources modified.
   It was difficult to track when and who made changes to the gateway and whether those changes had been deployed to the various stages.
4. The process felt slow and backward.
   Clicking a series of buttons to add something so fundamental to our system seemed like the wrong way to go about things. As we were building out the API Gateway resource-by-resource, we quickly learned how soul-sucking it could be to spend so much time clicking buttons in a UI. While using the AWS CLI made for a better experience for engineers who were familiar with the setup, it remained difficult for newer engineers to go in and comfortably make changes.

A solution for managing our API Gateway therefore needed to satisfy a couple main requirements:

1. Fine-grained control and clarity into the resources created.
   If creating a resource necessitated updating a policy document somewhere, we should be able to do exactly that.
2. Provide a changelog and versioning.
   Just like our application code, we should be able to see what changes were made at different points in time. Ideally, whatever solution we found would also provide state management and idempotency. A series of shell scripts, for instance, would not satisfy this requirement.
3. Empower other software engineers to make changes.
   This is the most important requirement. The API Gateway should be as self-serve as possible; engineers from feature teams should feel safe and confident when making changes to it. While the Platforms team at Apartment List is the custodian of the gateway, all other engineers should be able to contribute to it too!

Adopting Terraform

Terraform as an answer to our problems came to us in the form of a riveting lunch conversation our staff engineer Rick had with a former colleague who had been using it extensively at work. Despite it all making sense now in retrospect, it hadn’t occurred to us to use Infrastructure as Code since we had been light on DevOps-y projects at Apartment List. While exploring our options, we also evaluated Cloudformation as a prospect but ultimately chose Terraform because it’s platform agnostic and open source. There was also an ulterior motive: We named our API Gateway “Bifrost”, and getting to say “I’m Terraforming Bifrost!” sounded incredibly badass.

Terraforming our API Gateway turned out to be an amazingly smooth process. For each component, we declared corresponding resources inside our terraform files. Then, we ran terraform import to make our Terraform state aware of the existing resource and settled any discrepancies between our configuration and reality. Before we knew it, we had Terraformed Bifrost 🙌!

The next challenge to solve was a way to collaborate and verify changes before applying. Running terraform plan, verifying, then entering terraform apply on our own machines works well in a synchronous process or when few individuals actually run the Terraform commands, but it doesn’t scale well. In order for the gateway to be truly self-serve and available for all developers to safely make changes, we needed to introduce an additional tool or process to facilitate that. For that purpose, we found Atlantis. Using Atlantis enables us to give feedback and check configuration changes at the pull request stage, then apply changes before merging it in to master. This mirrors our regular dev process more closely by incorporating all the benefits of code review and leaves a log of changes applied in Github rather than an engineer’s terminal output.

Embracing Terraform

We’ve since started using Terraform to manage increasingly more resources at Apartment List, and so far we’re really happy with it. We’re excited to continue to explore new ways to improve our dev experience while setting ourselves up for growth. If you’re interested in being part of that journey, head on over to our careers page! We’d love to have you on the team.