Run multiple Postman API collections with Newman in parallel with automated HTML report generation using a one-line command

There are a lot of articles here and on the web related to the pros and contras of using the Postman for API testing, especially when it comes to the CICD integrations and test code maintenance.

I would like the main points to refresh the problems we — automation testers have.

While Postman is a popular tool for API testing, it does have some drawbacks, including:

1. Vendor Lock-In:

• Postman uses its own proprietary format for storing API tests, environments, and collections. This can lead to vendor lock-in, as migrating to another tool may require recreating tests from scratch.
• Exported Postman collections may not be easily transferable to other tools, potentially limiting flexibility and hindering collaboration.

2. Limited Support for Dynamic Data:

• Postman has limitations when it comes to handling dynamic data in API requests. This can be a challenge when dealing with APIs that generate dynamic responses or require dynamic inputs.

3. Sequential Execution on CI/CD:

• Postman collections are designed to be executed sequentially, and running them concurrently can be challenging. This can impact the efficiency of test execution, especially when dealing with large test suites.
• Running collections in parallel is possible with third-party tools, but it may require additional setup and is not natively supported in Postman.

4. Dependency on the Postman Platform:

• Postman relies on its cloud-based platform for features such as team collaboration, version control, and monitoring. This dependency may pose challenges for teams that prefer to use their own infrastructure or have specific security and compliance requirements.

5. Scripting Limitations:

• While Postman supports scripting using JavaScript, it may not be as powerful or flexible as using a dedicated programming language. This can limit the complexity of test scenarios and the customization of test logic.

6. Limited Integration with Source Control:

• While Postman does provide some integration with version control systems, the support might not be as seamless as desired. Teams may encounter challenges with collaboration and versioning when working on Postman collections in a shared environment.

7. Limited Reporting and Analytics:

• Postman’s reporting and analytics capabilities are somewhat limited. Teams may need to export and process test run results externally for more in-depth analysis and reporting.

If you have a complicated and large regression API test suites, that you need to Integrate to CI with Newman, providing the HTML reporting, I would like to share the solution I developed to bypass some of the Newman/Postman limitations.

During my journey as a Quality Assurance specialist, I faced multiple times frustrating situations with the Postman limitations for the automated API tests integration to the CICD pipelines.

By default, you have to write separate commands for each Postman collection you want to run with the Newman library, specifying the collections, environments, reporters, and export options for them.

This can quickly become quite messy and require regular maintenance and checks.

By the way, the Postman does not provide the universal possibility to run the collections in parallel without the need to have a separate command for each collection.

That’s exactly why we have a long story with the comparisons of the Postman with the Pytest or other API testing libraries like in these series
https://medium.com/exness-blog/api-testing-showdown-postman-vs-pytest-part-1-919a3a21a085

I would like to provide a simple solution for the issue explained in this StackOverflow conversation.

Imagine you have 20 collections in your API regression with 10 to 20 requests each.

Running it with the Newman in CICD, let’s say the Bamboo server will take around 7–10 minutes, which is quite long for the API tests.

The way to bypass the consecutive run and make it run in parallel will require writing the custom JavaScript code with the the help of ‘async’ package, which will run the same collections in just 1 minute, or the time the biggest collection takes.

I have implemented the standalone NPM package to handle this problem along with the reports generation:

• Framework will read all the collections in the destination folder, and filter them by specified name/env variable, with/without the use of an environment file from the specified folder.
• Junit report is generated for the CICD test results parsers
• HTMLextra and Allure reports are generated to share with the stakeholders
• You just need to specify a one-line command to run all/selected collections with/without the environment file in parallel

“Newman Parallel” is an npm package that executes Postman collections in parallel, saving time by running them concurrently. It supports reading collections and environments from separate folders and provides flexibility through command-line arguments.

The package is regularly maintained, adding new functionalities and updating the dependencies.

You can find the source code at https://github.com/Valiantsin2021/Newman-parallel-run.

The package has integrated Newman, Allure (allure results generated automatically), HTMLextra reporter, and Junit for CI.

HTMLextra and Junit reports will be generated in the “report” folder. The allure report will be generated in the “allure-report” folder after all the collections have run.

For the Allure report the command “npx allure generate — clean && npx allure-patch ./allure-report && rm -rf ./allure-report” will run automatically after all the collections have run.

It will generate the Allure HTML report patch it as a single page app to be sent via email, webhook, or published as a static webpage.

Allure HTML report with history will be generated only if you use GitBash/Bash. If you run the tests in Powershell the history will not be generated automatically.

Installation

npm install -g newman-parallel

Usage

newman-parallel [options]

Command-Line Options

• path: (Required) Path to the folder containing Postman collections.
• path: (Required) Path to the folder containing Postman environments.
• name: (Optional) Name of the collection/s to filter the collections.
• name: (Optional) Name of the environment to use.
• ALL: (Optional) to run all the collections from the folder

Environment variables

• the process logic will check the environment variables if there is no Name (C=<name>) of the collection/product passed in arguments
• If the environment variable/s equal to the name of the collection’s filename is set to ‘True’, the framework will run the collection/s that match.

Examples

Run all the collections that match the “MyCollection” with no environment used:

newman-parallel /path/to/collections /path/to/environments C=MyCollection

Run collections for a specific name with a specific environment:

newman-parallel /path/to/collections /path/to/environments C=MyCollection E=MyEnvironment

Run collections with a specific environment file(or without) depending on the environment variables specified:

# this will run collections that have XXX and YYY names

# GitBash/Bash
export XXX=True
export YYY=True
newman-parallel /path/to/collections /path/to/environments E=MyEnvironment
# or without the environment flag passed in arguments
newman-parallel /path/to/collections /path/to/environments

Run all the collections with a ALL argument passed (with or without environment):

# this will run all collections without environment file

newman-parallel /path/to/collections /path/to/environments ALL

# this will run all collections with specified environment file

newman-parallel /path/to/collections /path/to/environments ALL E=MyEnvironment

Notes

If no collection/product name and ALL arg is provided, the script runs all collections in the specified folder. If no environment name is provided, the script does not use any environment. An Allure and HTML extra reports will be generated automatically after the tests finish

Contributing

Feel free to write your opinions, and all Contributions are welcome!