Measure your code coverage using Istanbul (with a demo)

As part of my team’s latest effort to implement a full CI/CD process for our services, we sought to improve our confidence and trust in the changes we deliver. The first step was to invest some effort into making our product fully tested, and to significantly increase our code coverage. To this end, we’ve started looking for a code coverage tool for our existing and future Node.js services.

Before I describe the journey, a bit about code coverage.

Why code coverage?

It’s well understood that unit testing improves the quality and the maintainability of source code. BUT… how can we gauge the quality of our unit tests? Enter: code coverage measurement tools!

Essentially, code coverage is a simple measurement of the percentage of our source code that was executed during a cycle of automated tests. The code coverage score will be as high as the percentage of code that has been tested. Code coverage percentage is collected by using a tool which uses tracing calls and runs a full set of automated tests against the code. A good tool will tell you not only the percentage of the code that is executed, but also will allow you to drill into the data and see exactly which lines of code were executed during a particular test.

Code coverage VS Test coverage

To reiterate, code coverage measures and reports the number of executed code lines covered by automated tests. On the other hand, test coverage (AKA functional coverage) measures and reports the effectiveness of automated test cases in a qualitative manner; it measures how many of a system’s functionalities are covered by “black-box” automated test cases against the system.

Therefore, it’s important to remember: Code coverage isn’t directly indicative of code quality. High code coverage doesn’t guarantee a high test quality. It’s only a guarantee that some percentage of lines/paths/branches of code was executed. It doesn’t mean that those lines/paths/branches were tested in every configuration/situation. While code coverage can be used with any type of testing, it is most common and useful to tie it to unit tests.

The tool

In our pursuit of CI/CD, we defined a set of requirements that our tool of choice would need to meet. Some examples of our requirements are as follows:

• Has smooth integration with our test libraries (Jasmine and SuperTest).
• Has integration with our CI tool (TeamCity).
• Easy to configure.
• Provides visual and informative reports.
• Provides ability to enforce minimum code coverage threshold.

After some research, we found several good coverage tools and determined that Istanbul was the most suitable. Why? Istanbul provides simple integrations to Jasmine and TeamCity, easy configuration, visual coverage reports, and cool and informative drill-down options.

We included Istanbul as part of our internal Node.js micro-service template, which means that all the new micro-services will have built-in Istanbul integration and configuration (the threshold was set to 90%). The drilled-down and visualized reports made the code coverage process much more fun and interesting. The developers started to gamify it and compete for the most covered service. In a really short period of time, we’ve succeeded to significantly improve our micro-services’ code coverage and establish a great infrastructure for the entire R&D department.

TeamCity Integration

Demo Time!

Let’s move forward and build a simple project which demonstrates Istanbul’s integration and abilities. In this project we will build a full-coverage calculator web service in Node.js. We will use Jasmine, SuperTest libraries and play with the coverage reports (You can find the full project here).

Project initialization

1. Create a folder: mkdir istanbul-test-coverage-example
2. Navigate to the folder: cd istanbul-test-coverage-example
3. Initialize npm: npm init -> go through the process
4. Create the index.js file: dir > index.js
5. Install dependencies: npm install — save body-parser express
6. Install dev dependencies:

npm install -D jasmine jasmine-node istanbul jasmine-console-reporter supertest jasmine-supertest

1. Add the run scripts to the package.json

Calculator services creation

Now we are going to add all the web service files.

1. Add the following configuration and routes to the index.js file:

2. Create a new folder called services and implement the calculator functionality
(see example for adder.js service):

talkaminsky/istanbul-code-coverage-example

Run calculator service

npm run start

1. Open browser and go to: http://localhost:3000/adder?numberA=2&numberB=3

2. Verify that you get a response with the result.

Configure Jasmine

1. Create a new folder called spec.
2. Add a new file, specs.js with the following content:

3. Add a new file, jasmine.json with the following content:

Add unit tests to our services

1. Create a new folder called unit under specs.
2. Add a new file for each service and add your unit tests (see example for divider.spec.js):

talkaminsky/istanbul-code-coverage-example

Verify our coverage

1. Verify that test-cover run script exists:

"test-cover": "istanbul cover --include-all-sources spec/specs.js && istanbul check-coverage"

2. Run Istanbul:

npm run test-cover

3. Open Istanbul coverage report:

./coverage/lcov-report/index.html

The HTML page report

4. Drill down to identify the uncovered rows:

Drill down

5. Add tests in order to cover the code (see example for divider.spec.js):

6. Enforce code coverage by adding the following parameters to the end of the test-cover run script.

--statements 90 --functions 90 --lines 90

Add routes tests using supertest

1. In the spec folder create the file index.spec.js with the following contents:

Complete our coverage

1. Run Istanbul:

npm run test-cover

2. Open Istanbul coverage report again:

./coverage/lcov-report/index.html

Full coverage

After a great effort by my team, we’ve succeeded to fully cover our micro-services and decrease our number of bugs to 0.

Done!

If you love this post feel free to share and give 👏