Importance of JavaScript Test Coverage using Istanbul

Imagine you started working in a massive codebase. How confident would you be about changing a single line of code without it breaking any other lines? Probably not confident at all. Now, if this codebase has tests that all pass with your code change you would probably feel a little better. However, it only takes a single line of code to crash the application. Do we know that those existing tests run every line your code might affect? It happens all the time in the software development world where tested code still crashes in production because the tests only cover parts of the code or only some cases.

If you read this article, I assume you already know the importance of tests. However, writing tests does not guarantee a stable application. All tests might be passing, but the application might be still broken. That’s why it is important to make sure all lines of code are being run.

Can you imagine if there is a tool that would check what lines of code are not tested? It would lead to better applications and make the lives of developers easier.

Of course, there are tools like this! But let’s talk about the definition of test coverage first.

Definition of test coverage

Test coverage is a measure in percentage to check how much code of a program is being ran by the tests. It is done to lower the chance of leaving unnoticed bugs, detect the lines of code in the program that are not tested, identify useless tests, and notice the tests that need to be adjusted based on recent code changes.

Formula to measure test coverage:

lines_ran_by_test / total_number_of_lines * 100

If you get 100%, congratulations! You covered all lines of your code!

Coverage measurement tools

Jest is one of the most popular testing platforms that could be used with any JavaScript library or framework.

Istanbul is another popular all-JavaScript library testing tool that checks test unit coverage.

Coverage measurement tools like Istanbul and Jest are integrated with Mocha, Jasmine, Chai, SuperTest, and other testing frameworks to write test specs and then run Istanbul or Jest to calculate the percentage of the coverage.

In this article, I will walk you through an example of Istanbul usage with nyc because of its easy configuration and interesting coverage visuals. “Nyc” is Istanbul’s state of the art command-line interface. We will be using it to run Istanbul and generate our coverage reports.

Let’s begin!

1. Create your project.

mkdir TestArticleDemo
npm init

2. Install Mocha, Chai, Istanbul, and nyc as devDependencies.

npm install --save-dev mocha chai istanbul nyc

3. Add test scripts.

...
"scripts": {
   "test": "mocha",
   "test-with-coverage": "nyc --reporter=text mocha"
},
...

Nyc uses text reporter by default. Mocha will allow us to see the report in the terminal.

4. Create index.js file

touch index.js

5. Create test folder

mkdir test

6. Add index.spec.js in test folder

touch index.spec.js

7. Write some code in index.js

1. Function capitalize will take in string and capitalize the first letter.
2. Function reverseString will take in string and return a reversed string. If the function is taking anything else except a string, it will return “It is not a string”.
3. Function randomNumber will take in min and max numbers and return a random integer in between min and max numbers.
4. Don’t forget to export each function!

8. Now in index.spec.js file, let’s write some tests for our functions

1. Import “expect” with Chai.
2. Import capitalize, reverseString, randomNumber functions from index.js file.
3. Describe capitalize function where it expects to take in a string and capitalize the first letter of the string. It will return an empty string if the function takes in anything else except a string.
4. Describe reverseString function where it expects to take in a string and returns this string backwards.
5. Describe a randomNumber function where it expects to take in two numbers(min and max) and returns a random whole number in between these numbers.

9. Now when we have functions and their tests, we can see how Istanbul, nyc, and chai work together!

npm test-with-coverage

The top part of this output is what mocha gives us and the bottom part is what Istanbul gives us. Mocha tells us that all of our tests are passing, but Istanbul tells us that the test coverage is 87.5%. We can also see that line 13 in index.js file is not tested.

Let’s look at line 13 of our index.js file!

Oh no! We forgot to test our else statement! Let’s fix it with adding an extra test on line 28.

Now let’s make sure the new test passes.

npm test

Oh! We have a Reference error! We would have not caught this bug if it wasn't for Istanbul. Let’s fix that real quick.

function reverseString(str) {
  if (typeof str == "string") {
    return str.split("").reverse().join("");
  } else {
    const error = "Error: ";
    return error + "It is not a string";
  }
}

That is what we meant to do in that else clause but since it was an edge case we didn't give it the time of day while testing.

Let’s run tests again after the fix!

Hooray! Now we got 100% of test coverage!

This was a simple example, but it would work the same way with thousands of lines of code where full coverage is very important to avoid bugs. It’s also easier to read the meaning of each function. Even without looking at my code, you can see what every function does just by looking at the tests’ description.

If you have thousands lines of code, there is also a nicer way to determine the lines of code that are missing tests.

1. Just add reporter=html in your script.

...
"scripts": {
  "test": "mocha",
  "test-with-coverage": "nyc --reporter=html mocha"
},
...

Now by running test-with-coverage script, it will create a coverage folder in your project. Index.html is the report file we will need to examine our test coverage. Copy its path and paste in your browser window!

Just like in the previous report when we were missing an else statement, it tells us the percentage of total coverage.

The script with reporter=html also lets us open index.js file and see the line we forgot to test!

The E on line 10 means that the else for this if is not ran.

Test coverage measurement tools could save you from a lot of hidden bugs and give you more confidence when deploying your code to production instead of randomly breaking.

What would it not fix?

Test coverage does not guarantee an application has no issues at all. Remember a test is only as good as the assumptions it is making. For example, a function helloWorld has to return “Hello World!” but returns “hello world!”.

function helloWorld(str) {
  return "hello world!";
}

We did not test if the first letters are capitalized but instead tested the length of a returned string.

describe("helloWorld function", function () {
  it("should return 'Hello World!' ", () => {
    const result = helloWorld("hello world!");
    expect(result).to.have.lengthOf(12);
  });
});

The test would pass and the line of the code is covered, though the application is technically not correct if the rest of the code is expecting capital letters or the actual requirement for the code is to have capital letters. We would need to re-write the test to test the main reason for the function.

The art of writing meaningful tests could be an entire article on its own.

describe("helloWorld function", function () {
  it("should return a string where each word is capitalized ",() =>{
    const result = helloWorld("hello world!"); 
    expect(result).to.equal("Hello World!");
  });
});

Conclusion

Tests and test coverage should exist together as important representatives of a stable application. In reality, a partially tested codebase is not far off from a completely untested codebase. The only way to gain real confidence in your code is to have 100% test coverage.