From manual to automated testing: The roadblocks and the journey

API developers and testers often take the manual testing route when it comes to debugging and testing their services. What are the challenges that they face in automating their API test suite? How can they begin their journey towards automation?

In my previous article, I talked about practices that can be implemented to build sustainable processes for integration testing of APIs and microservices. I had mentioned transitioning from manual to automated testing as one of the items on the checklist in that article. That journey requires more of a transformation than a transition.

Goals — Short term: Automation. Longer term: More Automation. Idea courtesy Kunal Nagpal.

We have been talking to engineers about how they build and consume services, both internal and external to their organization. We talked to system engineers, QA engineers, test automation experts, backend developers and full-stack developers. One of the recurring themes I found across all these conversations is when it comes to testing APIs they use a tool like Postman to debug their API — but, often, they do it manually.

Manual is not (always) productive

For a developer building a microservice, this means continuously using Postman’s requests and history to query the service they are developing, then watch the response to verify if they received the desired data. For a test engineer, it means something very similar, except they are usually given a service URL to interact with along with documentation in a hand-edited document that they have to test piece by piece. For a systems engineer building an application delivery pipeline, this amounts to querying multiple integration points by hand with disparate documentation and trying to get the deployment pipeline in place with proper dependencies. The scenario complicates based on how many other services they have to interface with.

While this kind of manual approach works in the shorter run to get things done, it is not scalable, adds too much personnel overhead and is not an enjoyable experience for the people involved.

“Is there any way I can be more productive if I use other features of your tool?”

In one of those sessions, an engineer authoring a microservice that was entirely meant for internal use in their company asked me, “I currently search through the history list [in Postman] to find requests that I make frequently. Then I edit the parameters and send a new request. Is there any way I can be more productive if I use other features of your tool?”

That conversation showed the need to increase productivity through better tooling to automate the boredom of the daily routine. The ask was not necessarily about spending less time, but spending more quality time in their day to day job.

The divide

Postman is a complete API development environment with features like scripting, automated API documentation, mock servers and monitors. But a section of our user base start by using Postman as a manual testing tool. This gap exists even after many teams have successfully integrated Postman with other testing tools and have built test automation suites. Beyond Postman, there are other tools out there to automate API testing that a team can put together to build an automation process. Plus, there are public documentation, tutorials, courses, books, articles, talks and workshops on testing automation.

Still, why do so many members of the tech ecosystem struggle to upgrade from manual to automated testing?

I think we, the tech community, have offered a plethora of solutions, but we haven’t taken the time out to hear and understand the pain points that our peers have. We need to do more to reach out to those in the industry, whether new or veteran, who would benefit from a small conversation or need an overview into how all the pieces fit together.

I understand this pain first-hand. I have gone through phases of learning where the primary challenge was not in finding resources, but in finding the right resources that fit together in the right order to give me a holistic understanding of subject I tried to learn.

The main roadblock on the journey from manual to automated testing is the lack of clear understanding of how the whole process would work end to end.

Postman Collections are reusable test suites

Let us address some of these missing links here and see where Postman fits in.

If you are a developer building microservices or an engineer building a tool for in-house usage, or if you are part of a team that is working on delivering an API, Postman’s Collections can help you in reducing the amount of manual tasks you do every day. Collections let you organize requests you make in Postman. They also form the basis of other useful features like mocks, monitors and documentation.

Stages of adding requests in a collection to build a reusable test suite

You can write tests in Postman for each request to assert on the response received using scripts. The scripts you write can control the workflow and order in which requests are executed. You can also use popular libraries like MomentJS or Lodash in Postman scripts.

When you save these requests to a collection and organize them in folders, you end up with an executable, reusable and collaborative test suite for your APIs.

Looking at existing collections

The Postman API network has some excellent collections published by other companies.

The Postman API Network is used to publish your API’s collections publicly.

Those published by Box, Auth0 and Imgur are great reference points for anyone trying to publish their collections, whether internally or externally. You can scroll through their documentation and use the “Run in Postman” buttons on those pages to load the respective collections into the Postman App.

Imgur provides a collection with tests so you can trigger a collection run for the whole test suite. Here is how the Imgur documentation looks:

The Imgur API documentation includes a “Run in Postman” button.

Auth0’s collections document endpoints exposed by their APIs. The main value of their collection is to serve as documentation for developers building custom integrations in their app with Auth0’s API. This is how the Auth0 documentation talks about using Postman for testing the collections they have built:

Auth0’s API documentation page where they talk about testing Auth0 APIs with Postman

Run collections anywhere

Once you have built a collection, you can execute it to run the requests saved in that collection in one go. This is synonymous to executing a traditional test suite, except your test suite is now a living documentation of your API.

You can run collections directly from the Postman app. You can run them in the Postman cloud with Monitors. You can also run collections on the commandline with newman. If you are building a CI pipeline or already have one, you can trigger collection runs with each build using newman.

Collections can be run from the Postman app, in the Postman cloud or using the newman commandline tool

Running locally

When you are developing a service on your local machine or testing it locally, you can trigger collection runs using the Postman app as you progress. That way you don’t have to build individual requests every time you test, but have the whole suite run in the sequence of your choice.

You can use variables and environments to reuse the same collection in different machines. For example, a common practice is to use a {{base_url}} placeholder in your request URL and switch environments to hit either a locally running server or a serve running on a remote machine.

As you keep grouping requests into folders in the collection, you can optionally run only a subset of the collection by selecting a folder when triggering a collection run. This lets you avoid running the whole suite when you want to test only a specific group of endpoints.

Running in CI

newman can run a collection either by the URL of the collection or by pointing to a locally available collection definition. You can inject environment variables and data files as part of a newman run, just like you do it in the app. Depending on how your CI pipelines are set up, there are a few ways to set up newman in your continuous integration process:

• Have your CI scripts fetch a collection using the Postman API and then run it using newman.
• Inject the URL to your collection as part of an environment variable in your CI runs. If your CI system runs completely on a private, protected infrastructure, you would need to whitelist Postman’s domains to let newman access them.

Exporting a Collection to JSON

Exporting an environment to JSON

newman can produce different types of reports, including HTML and JSON. You can save these reports as part of your build artifacts to access them after the build is over.

Run the collection on commandline with newman with the right environment

For more, look at the articles on how to integrate newman with Jenkins, or with Travis CI. You can also run newman in a Docker container. Plus, our lovely community publishes resources that they have found useful:

Automate, even before you start developing

You can start automating right from the design phase of your APIs. Build a separate collection that documents all the required endpoints in your collection. You can then use that collection to create mock servers for your developers and your testers. That was the core argument of the first article in this publication, “Fake it till you make it: mocks for agile development”.

So, as a QA you can start writing your test suites against expected responses from the mock. As a frontend engineer, you can start building your apps without waiting for your backend engineers to build out the service.

You don’t even need to flesh out a continuous integration system from the beginning. You can use Postman Monitors to keep running your collections periodically. Each monitor run executes the tests that are part of requests in your collection, giving you a good view of the freshness of your collection.

Collaborate. Better.

Postman teams are now available in the free plan. You no longer need to wait to have a Pro account to try out what seamless collaboration can be like. Once you start making use of teams and workspaces to organize, share and edit your collections, you make collaboration a first class citizen of your development and testing process.

You don’t need to manually send your API documentation around, or share API specifications over email or a wiki. You build a shared, single source of truth.

In the end, you end up automating not just your tests but your workflow as well. With a little upfront planning and with the right tools, your journey from manual to automated testing of APIs becomes a shared story. You free up resources that would have otherwise wasted in manual tasks. Those resources would then be available for you to spend on more productive outcomes — as an individual or as a team.

At Postman, we try to solve gaps in development workflows by building tools that can make software development better, more efficient and enjoyable. Watch this publication for more such articles and tweet @postmanclient to let us know how you use Postman in your daily work routine!