How We Built a Self-Service API Testing Platform in AWS

Snag is a platform for hourly work, so naturally our focus is on connecting businesses and employers and empowering them with tools like on-demand shifts, chatbot-based hiring, and awesome matching algos. The engineering org that supports this tech stack uses a model where product teams each have a “Quality Engineer”, who writes automated tests. And as you can imagine, a stack this diverse makes for some interesting scenarios.

The Problem

While we had achieved our goal of establishing a robust and extensible web testing framework, we had a couple of issues:

1. Greenfield development. Our devs were pumping out new UI features faster than QE’s could write tests.
2. Regression issues. New features were fine, but we were breaking older stuff with each deploy.
3. UI tests suck. They’re (1) time-consuming to create; (2) difficult to maintain; and (3) are costly to execute. Also, at the time, they were failing irregularly due to elusive networking issues related to a third-party browser virtualization service.

In addition to these challenges, there was some extra context worth mentioning:

• Aggressive timeline. To stay ahead of the competition, slowing down was not an option.
• Evolving product. Ongoing beta testing feedback meant changing front-end flows, which means rewriting UI tests.
• QE culture. Dedicated QE’s are responsible for writing/maintaining code for automated testing.*

*A note on QE culture: it isn’t necessarily a bad thing, but we wanted to move away from Dev & Ops & QA silos and towards a self-service model to encourage a sense of shared ownership of quality.

Why API Testing?

After identifying the issues and factoring in the context, it became clear that, while UI testing was irreplaceable and needed to be fixed somehow, it was the right time to build out an API testing platform for three (3) main reasons:

1. Simple to write. It’s just a simple cURL with a check for 200 OK, with some basic environment and variable management made easy with Postman.
2. Lightning fast to execute. Like 200ms fast.
3. Easy to maintain. Schemas rarely change.

For us, API testing offered us the ability to set up a basic, full-body health check on our app, so devs could keep releasing code and our QE’s could focus on more complex UI scenarios. And, being lightweight & durable, it was a perfect candidate for CICD integration.

The Success Criteria

So we came up with a high-level “solution statement” for a self-service API testing prototype that we could achieve within 6 to 12 months:

1. Automated testing of API’s against any integration environment.
2. QE ownership of a results display, data store, and infrastructure.
3. Bonus: Developers can maintain it. Extra bonus: PO’s can too.

The Strategy

Next, we drafted a 10,000-foot approach. In essence, we would:

1. Explore API testing. Make a list of options & rank them. Develop prototypes for the top 3 & score them against key success criteria. Pick a solution and build consensus as a QE chapter.
2. Implement it on 1 product team. Find a product team that’s eager to adopt the tool into their workflow. Solicit feedback and tweak. Measure success.
3. Roll out org-wide. Bring in Engineering leadership and hammer out a plan for adoption.

The Solution

To start, we decided on three (3) key AWS tools that we would design around:

1. ECS. Offered QE ownership of the service via Docker.
2. SQS. Enabled tests to trigger from a Lambda job, which would hang off every new service deploy
3. Newman. Lightweight open-source test runner for Postman collections, abstracting the execution and providing an awesome UI for editing tests.

At a high level, ECS deploys would trigger messages onto an SQS queue…

…and our service would pick them up, execute the tests with Newman, and export to Dynamo (& Slack).

For glue, we used the following secondary technologies:

• Ruby. For scripting and most logic, we used Ruby because it’s powerful and we already knew it well. Also:

• Postman Pro API. Developers already know and love the free version. Pro API allows devs to easily maintain auto-saving collections on a beautiful UI. Also, it’s cheap and easy to replace if the price goes up :)
• DynamoDB. We don’t need complex indexing/searching, so DynamoDB is perfect for exposing simple datasets to a basic results UI.

For effect, here’s how things looked when you put them all together:

Some important details:

1. Developers create and maintain their Postman collections and environments on their local machine (automatically stored in Postman API) and are responsible for maintaining the correct service-name:guid key-value pair within Consul for the collection they want to run.
2. The postman-api service is running on each environment’s ALB, just like the other services.
3. At runtime, the service looks in Consul for a collection guid matching it’s name, grabs the guid along with a secret Postman token, and provides Postman Pro API with the guid and token in exchange for the test collection, which is served in JSON.
4. The postman-api service is running on each environment’s ALB, just like the other services.
5. The details on the SNS message are passed via Lambda from the service that’s being deployed

1. Anything sensitive (Postman guid/token/secretkey, etc.) is stored in Consul under a shared Postman directory
2. Results are exported to .json, parsed into a Ruby object, exported to DynamoDB (which is indexed into a single results UI). Extra logic pings a specified Slack channel if there’s a failure with a link to the results for that specific text run.

Next Steps

In the coming months, we will refine this design as developers begin using the tool and providing feedback. As I said before, the goal is developer empowerment. That is, as QE’s we’re focused on providing tools that give developers the power to test their changes. And the point of that is better features delivered faster, which means more magical moments where the right workers meet the right employers who need them. :)