From manual to automated testing: The roadblocks and the journey

API developers and testers often take the manual testing route when it comes to debugging and testing their services. What are the challenges that they face in automating their API test suite? How can they begin their journey towards automation?

Kaustav Das Modak
Sep 5, 2018 · 8 min read

In my previous article, I talked about practices that can be implemented to build sustainable processes for integration testing of APIs and microservices. I had mentioned transitioning from manual to automated testing as one of the items on the checklist in that article. That journey requires more of a transformation than a transition.

Goals — Short term: Automation. Longer term: More Automation. Idea courtesy Kunal Nagpal.

We have been talking to engineers about how they build and consume services, both internal and external to their organization. We talked to system engineers, QA engineers, test automation experts, backend developers and full-stack developers. One of the recurring themes I found across all these conversations is when it comes to testing APIs they use a tool like Postman to debug their API — but, often, they do it manually.

Manual is not (always) productive

While this kind of manual approach works in the shorter run to get things done, it is not scalable, adds too much personnel overhead and is not an enjoyable experience for the people involved.

Is there any way I can be more productive if I use other features of your tool?

In one of those sessions, an engineer authoring a microservice that was entirely meant for internal use in their company asked me, “I currently search through the history list [in Postman] to find requests that I make frequently. Then I edit the parameters and send a new request. Is there any way I can be more productive if I use other features of your tool?

That conversation showed the need to increase productivity through better tooling to automate the boredom of the daily routine. The ask was not necessarily about spending less time, but spending more quality time in their day to day job.

The divide

Still, why do so many members of the tech ecosystem struggle to upgrade from manual to automated testing?

I think we, the tech community, have offered a plethora of solutions, but we haven’t taken the time out to hear and understand the pain points that our peers have. We need to do more to reach out to those in the industry, whether new or veteran, who would benefit from a small conversation or need an overview into how all the pieces fit together.

I understand this pain first-hand. I have gone through phases of learning where the primary challenge was not in finding resources, but in finding the right resources that fit together in the right order to give me a holistic understanding of subject I tried to learn.

The main roadblock on the journey from manual to automated testing is the lack of clear understanding of how the whole process would work end to end.

Postman Collections are reusable test suites

If you are a developer building microservices or an engineer building a tool for in-house usage, or if you are part of a team that is working on delivering an API, Postman’s Collections can help you in reducing the amount of manual tasks you do every day. Collections let you organize requests you make in Postman. They also form the basis of other useful features like mocks, monitors and documentation.

Stages of adding requests in a collection to build a reusable test suite

You can write tests in Postman for each request to assert on the response received using scripts. The scripts you write can control the workflow and order in which requests are executed. You can also use popular libraries like MomentJS or Lodash in Postman scripts.

When you save these requests to a collection and organize them in folders, you end up with an executable, reusable and collaborative test suite for your APIs.

Looking at existing collections

The Postman API Network is used to publish your API’s collections publicly.

Those published by Box, Auth0 and Imgur are great reference points for anyone trying to publish their collections, whether internally or externally. You can scroll through their documentation and use the “Run in Postman” buttons on those pages to load the respective collections into the Postman App.

Imgur provides a collection with tests so you can trigger a collection run for the whole test suite. Here is how the Imgur documentation looks:

The Imgur API documentation includes a “Run in Postman” button.

Auth0’s collections document endpoints exposed by their APIs. The main value of their collection is to serve as documentation for developers building custom integrations in their app with Auth0’s API. This is how the Auth0 documentation talks about using Postman for testing the collections they have built:

Auth0’s API documentation page where they talk about testing Auth0 APIs with Postman

Run collections anywhere

You can run collections directly from the Postman app. You can run them in the Postman cloud with Monitors. You can also run collections on the commandline with newman. If you are building a CI pipeline or already have one, you can trigger collection runs with each build using newman.

Collections can be run from the Postman app, in the Postman cloud or using the newman commandline tool

Running locally

You can use variables and environments to reuse the same collection in different machines. For example, a common practice is to use a {{base_url}} placeholder in your request URL and switch environments to hit either a locally running server or a serve running on a remote machine.

As you keep grouping requests into folders in the collection, you can optionally run only a subset of the collection by selecting a folder when triggering a collection run. This lets you avoid running the whole suite when you want to test only a specific group of endpoints.

Running in CI

  • Have your CI scripts fetch a collection using the Postman API and then run it using newman.
  • Inject the URL to your collection as part of an environment variable in your CI runs. If your CI system runs completely on a private, protected infrastructure, you would need to whitelist Postman’s domains to let newman access them.
Exporting a Collection to JSON
Exporting an environment to JSON

newman can produce different types of reports, including HTML and JSON. You can save these reports as part of your build artifacts to access them after the build is over.

Run the collection on commandline with newman with the right environment

For more, look at the articles on how to integrate newman with Jenkins, or with Travis CI. You can also run newman in a Docker container. Plus, our lovely community publishes resources that they have found useful:

Automate, even before you start developing

So, as a QA you can start writing your test suites against expected responses from the mock. As a frontend engineer, you can start building your apps without waiting for your backend engineers to build out the service.

You don’t even need to flesh out a continuous integration system from the beginning. You can use Postman Monitors to keep running your collections periodically. Each monitor run executes the tests that are part of requests in your collection, giving you a good view of the freshness of your collection.

Collaborate. Better.

You don’t need to manually send your API documentation around, or share API specifications over email or a wiki. You build a shared, single source of truth.

In the end, you end up automating not just your tests but your workflow as well. With a little upfront planning and with the right tools, your journey from manual to automated testing of APIs becomes a shared story. You free up resources that would have otherwise wasted in manual tasks. Those resources would then be available for you to spend on more productive outcomes — as an individual or as a team.

At Postman, we try to solve gaps in development workflows by building tools that can make software development better, more efficient and enjoyable. Watch this publication for more such articles and tweet @postmanclient to let us know how you use Postman in your daily work routine!

Better Practices

For individual engineers to the largest teams, Better Practices is intended to distill knowledge from the Postman community. This is a place to learn about modern software practices together! Read more:

Thanks to Abhinav Asthana

Kaustav Das Modak

Written by

Developer Advocate at Postman. Looking for patterns in chaos.

Better Practices

For individual engineers to the largest teams, Better Practices is intended to distill knowledge from the Postman community. This is a place to learn about modern software practices together! Read more:

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade