Newman Integration in API Automation with Postman, Html Reporter, and CI/CD Pipeline
Hi everyone,
In this article, I will share the processes I’ve experienced with Newman integration, Html reporter, and CI/CD pipeline in API automation via Postman.
This article will consist of three steps. First of all, we will run our tests from the command line with the help of Newman, which we got used to running from the Postman interface before. Then we will take it one step further and create more readable and shareable HTML reports of our tests. At the end, we will reward all these processes with a CI/CD integration.
I hope you will enjoy reading this article.
Newman
Newman is an npm(Node Package Manager) technology that allows us to run and test Postman collections directly from the command line rather than Collection Runner. Npm is a package management system developed for the javascript programming language and accepted as a standard by Node.js. It can be run from the Npm command line. Thus, CI (Continuous Integration) can be easily installed on systems and servers (CircleCI, Jenkins, Travis CI, Gitlab, Bamboo). At this place, Newman can be imagined as a gateway to other tools.
You can complete the Newman integration by following these steps below.
- In order to run Postman Collections with Newman from the command line, you must first download and install node.js on your machine. Download and install node.js from https://nodejs.org/en/.
After the installation has been completed, you can verify your npm version by typing the following command in the terminal.
$ node -v
2. After installing Node.js, open the terminal. Install Newman with the following command below.
$ npm install –g newman
After the installation has been completed, you can check the version by typing the following command in the terminal.
$ newman -v
3. Finally, you are ready to run your tests via the command line. This run activity can be performed in 2 different ways.
- Export the collection which you will run and go to the directory where that file is located from the command line. Then you can test the collection by running the following command.
newman run ‘collection_name.json’
When you run the test, you will encounter a result similar to the below.
- You can also run the tests via the link generated for collection. In fact, this method is much more effortless than exporting. Because each change made in the collection will be automatically affected on the collection link. You won’t have to spend any extra effort for the changes. However, doing this by exporting will cost us to get a new export for each change.
newman run collection_link
When you run the test by typing the command above, we expect to see exactly the same result as the run performed by exporting. For sure, you can run the test in any directory since you provide the link.
Additionally, you can run ‘newman run –h’ to see which features you can use in Newman.
I mostly use these features among them are -n, -e and -g. These features are explained below.
- -n : It is the feature that we define how many times we will run the collection. For instance, in order to run 10 times;
newman run collection_name.json –n 10
- -e : It allows us to use the variables used in the collection and defined in the environment file as in the Collection Runner. For instance;
newman run collection_name.json –e environment.json
- -g : Allows us to use the variables used in the collection and defined in the global file. For instance;
newman run collection_name.json –g MyWorkspace.postman_globals.json
HTML Reporter
Newman also allows us to report the tests we run from the command line in HTML format. HTML reporter is quite convenient especially using CI server. This reporter comes with a dashboard style summary landing page and a set of different tabs which contain the detailed request information. It creates a much more detailed report than the console. It provides us to capture the body, header, params etc fields in detail for request and response. It helps us to debug failed tests and makes it easy for whole stakeholders in the project in order to review the test results with an overview.
You can complete HTML reporter installations by running the following instruction from the command line.
npm install –g newman-reporter-htmlextra
While running a collection, use the -r cli, htmlextra as in the command below. After the test is run, the HTML file will be automatically generated under the Newman folder in the related directory.
newman run https://api.postman.com/collections/6544200-6318cae5-da0c-45a1-b37f-841a91f6e957?access_key=PMAT-01GM0A5BE99ED1AVVCJ21G1EZV -r cli,htmlextra
The ‘cli’ that I mentioned above allows us to receive test results from the console. Besides, the ‘htmlextra’ allows us to get test results in HTML format. If our goal is to get just HTML reports, we may not use the cli command.
In this report, especially the Total Requests tab helps us a lot in big projects. When we enter any test in the report, a very detailed report about the request and response welcomes us. This is pretty helpful especially in cases when errors appear. Being able to see the following fields clearly and legibly makes it easier for us to reach a solution.
So, how about taking it one step further? Can we fulfill all these steps on a CI server? Thus, we can guarantee our service and process with automatic tests that run each time whenever the project is built.
In the remaining part of this article, I will touch on the CircleCI integration. Certainly, integrations for platforms such as Jenkins, Travis CI, Gitlab, Bamboo can be integrated by reviewing the relevant documentation and following the relevant steps.
CI/CD Integration
CircleCI is one of the most popular CI/CD platforms in the world. It simply automates the build, test and deployment processes for a software. CircleCI runs each job in a separate container or VM. So, every time the job is run, CircleCI creates a container or VM in order to run the job.
In CircleCI, all the magic is in the .circleci/config.yml file. We will perform all the configuration on this file.
First of all, we will try to run our tests in CircleCI using the postman/newman image. For detailed info about this subject, you can visit https://hub.docker.com/r/postman/newman/. In the light of the documentation at this address, the ‘Integration tests’ step in config.yml is prepared as below.
When we examine the run in CircleCI, we see a similar run in CircleCI that we perform locally. From now on, our tests are running in each development performed in the service and all stakeholders can reach the results in CircleCI. Otherwise, even if one of the tests gets an error, the relevant build will be broken. In this way, a failed code will be prevented from being released into the production environment.
We can now run our tests in CircleCI and report the results on a console screen. Well then, how can we see the results of this run as an HTML report?
Hereby, we will benefit from an open source project written using Handlebars and Javascript. You can find the github link of this reporting tool below.
https://github.com/DannyDainton/newman-reporter-htmlextra
By using dannydainton/htmlextra image, we can pull node, newman and htmlextra packages altogether. In this way, we can run our tests without adding extra steps to the config.yml file. You can see the DockerHub link of the related image below.
https://hub.docker.com/r/dannydainton/htmlextra
Also, this project provides various CLI options. For instance, one of the options we will use here is the ‘ — reporter-htmlextra-export <path>’ option. As seen in local runs, when a directory is not specified to newman, it saves the HTML file which it creates to the current working directory. Also, as seen in local runs, the name of the HTML file is created by adding the date info to the end of the collection name.
In summary, we will create a volume for reporting below. Then we will run our test and export the HTML report created in this volume with the name of ‘Integration.html’. Finally, we will copy the exported report to the ‘ARTIFACTS’ field under the relevant build.
Note: The build will be broken when the ‘Integration tests’ step gets an error. Also, I would like to focus on when: always command that is located under the ‘Html report exporting’ step. Normally, when the ‘Integration tests’ step gets an error, the build will be broken and the next steps will not continue. This means that if the test fails we will not be able to export html report. Thanks to this feature, even if the ‘Integration tests’ step has an error, the report will be copied under the ‘Artifacts’ field.
I would like to thank Büşra Karagöz and Musa Meriç for their support.
