An Overview of JavaScript Testing in 2017

This short guide is intended to catch you up with the most important reasoning, terms, tools, and approaches to JavaScript testing. It combines information from many great recently written articles and adds a little more from our own experience.

***** Important update: ****

A new version of this guide was released.

Look at the logo of Jest, a testing framework by Facebook:

As you can see, their slogan promises a “painless” JavaScript Testing, but as “some guy from the comments” said:

And indeed, Facebook have an excellent reason to use this slogan. In general, JS developers are not too happy with website testing. JS tests tend to be limited, hard to implement, and slow.

Nevertheless, with the right strategy and the right combination of tools a nearly full coverage can be achieved and tests can be very organized, simple, and relatively fast.

Here it is worth mentioning there are many great unmaintained libraries I came across when writing this blog post, with unique features that can potentially be very useful in various situations if a company decides to resurrect and maintain them such as- DalekJS.

I’ll hopefully create a summary of those in a future blog post, but for now, let’s focus on the trending maintained libraries.

Test Types

You can read about different test types in more depth here and here and here.
In general, the most important test types are:

• Unit Tests- Testing of individual functions or classes by mocking input and making sure the output is as expected.
• Integration Tests- Testing several modules to ensure they work together as expected.
• Functional Tests- Testing a scenario on the product itself (on the browser, for example) regardless of the internal structure to ensure expected behavior.

Test Tools Types

Test tools can be divided into the following functionalities. Some provide us with only one functionality, and some provide us with a combination.

It’s common to use a combination of tools even if one can achieve the same using a single tool in order to get a more flexible functionality.

1. Provide a test environment (Mocha, Jasmine, Jest, Karma)
2. Provide a testing structure (Mocha, Jasmine, Jest, Cucumber)
3. Provide assertions functions (Chai, Jasmine, Jest, Unexpected)
4. Generate, display, and watch test results (Mocha, Jasmine, Jest, Karma)
5. Generate and compare snapshots of component and data structures to make sure changes from previous runs are intended (Jest, Ava)
6. Provide mocks, spies, and stubs (Sinon, Jasmine, enzyme, Jest, testdouble)
7. Generate code coverage reports (Istanbul, Jest)
8. Provide a browser or browser-like environment with a control on their scenarios execution (Protractor, Nightwatch, Phantom, Casper)

Let’s explain some of the terms mentioned above:

Testing structure refers to the organization of your tests. Tests are usually organized in a BDD structure that supports behavior-driven development (BDD). It often looks like this:

describe('calculator', function() {
  // describes a module with nested "describe" functions
  describe('add', function() {
    // specify the expected behavior
    it('should add 2 numbers', function() {
       //Use assertion functions to test the expected behavior    
    })
  })
})

Assertion functions are functions that make sure tests result as expected where the most popular are the first two:

// Chai expect
expect(foo).to.be.a('string')
expect(foo).to.equal('bar')
// Jasmine expect
expect(foo).toBeString()
expect(foo).toEqual('bar')
// Chai assert
assert.typeOf(foo, 'string')
assert.equal(foo, 'bar')
// Unexpected expect
expect(foo, 'to be a', 'string')
expect(foo, 'to be', 'bar')

TIP: Here is a nice article about advanced Jasmine assertions.

Spies provide us with information about functions that are used in the application or created for tests- How many times they are called, in what cases, and by who? They are especially useful in integration tests where we want to ensure certain behaviors when running an internal scenario. For example, how many times was a calculation function called during some process?

it('should call method once with the argument 3', () => {
  const spy = sinon.spy(object, 'method')
  spy.withArgs(3)
  object.method(3)
  assert(spy.withArgs(3).calledOnce)
})

Stubbing or dubbing (like doubles in movies) replaces selected functions with our own to ensure an expected behavior on selected modules.

If we want to ensure user.isValid() always returns true during a test, for example, you can do this:

sinon.stub(user, 'isValid').returns(true) // Sinon
spyOn(user, 'isValid').andReturns(true) // Jasmine

This also works with promises:

it('resolves with the right name', done => {
  const stub = sinon.stub(User.prototype, 'fetch')
    .resolves({ name: 'David' })

  User.fetch()
    .then(user => {
      expect(user.name).toBe('David')
      done()
    })
})

Mocks or Fakes are faking certain modules or behaviors to ensure a test runs with known inputs. Sinon can, for example, fake a server to ensure fast and expected results.

it('returns an object containing all users', done => {
  const server = sinon.fakeServer.create()
  server.respondWith('GET', '/users', [
    200,
    { 'Content-Type': 'application/json' },
    '[{ "id": 1, "name": "Gwen" },  { "id": 2, "name": "John" }]'
  ])
  Users.all()
    .done(collection => {
      const expectedCollection = [
        { id: 1, name: 'Gwen' },
        { id: 2, name: 'John' }
      ]
      expect(collection.toJSON()).to.eql(expectedCollection)
      done()
    })

  server.respond()
  server.restore()
});

Snapshot Testing is when you compare a resulted data structure to an expected one. For example, the following Jest test simulates the rendering of a “Link” component and then saves it as JSON.

It then compares it to the results of a previous run. If anything changed, the developer is prompted to agree that the change is intended.

it('renders correctly', () => {
  const linkInstance = (
    <Link page="http://www.facebook.com">Facebook</Link>
  )
  const tree = renderer.create(linkInstance).toJSON()
  expect(tree).toMatchSnapshot()
})

Putting it All Together

We suggest using the same tools for all the test types if possible. The same testing structure and syntax (2), assertion functions (3), result reporting, and watching (4) for all of them. Sometimes even the same testing environment (1) can be used for some or all of them.

Make sure it is possible to run only particular types of tests on demand.

• For unit tests, provide all units with a mocked input (6) and make sure their output is expected (3), Also make sure to use a coverage reporting tool (7) to know what units are covered.
• For integration tests, define important cross-module internal scenarios. Comparing to unit tests, you would use spies and stubs to test expected behavior instead of just asserting the output (6). Also, a browser or browser-like environment could test integration between processes and their results on the UI.
• For functional tests, A browser or browser-like environment with a programmable API (8) will be used to create user-like behavior scenarios.

***** Important update: ****

A new version of this guide for was released.

List of General Prominent Testing Tools

node-jsdom

JSDom is a JavaScript implementation of the WHATWG DOM and HTML standards. In other words, JSDom simulates a browser’s environment without running anything but plain JS.

In this simulated browser environment, tests can run really fast. The drawback with JSDom is that not everything can be simulated outside a real browser (you can’t take a screenshot for example) so using it will limit your tests’ reach.

It is worth mentioning that the JS community rapidly improves it.

istanbul-js: A Javascript code coverage tool written in JS

Istanbul will tell you how much of your code is covered with unit tests. It will report on statement, line, function and branch coverage in percentages so you will understand better what is left to cover.

PhantomJS

Phantom implements a “headless” Webkit browser which is between a real browser and JSDom in speed and stability.

It is very popular in the time of writing this article, but since Google are adding the feature of running “headless” to the native Google Chrome browser, it is not longer maintained by it’s main creator and maintainer Vitaliy Slobodin.

karma-runner/karma

Karma lets you run tests in browsers, including real browsers, Phantom, jsdom, and even legacy browsers.

Karma hosts a test server with a special web page to run your tests in the page’s environment. This page can be run across many browsers.

This also means tests can be run remotely using services like BrowserStack.

chaijs/chai

Chai is the most popular assertion library.

unexpectedjs/unexpected

Unexpected is an assertion library with a slightly different syntax from Chai. It is also extensible so assertions can be more advanced with libraries that are based on it like unexpected-react that you can read about more in depth here.

Sinon.JS - Standalone test spies, stubs and mocks for JavaScript. Works with any unit testing framework.

Sinon is a very powerful standalone test spies, stubs and mocks for JavaScript that works with any unit testing framework.

testdouble.js

testdouble is a new library that is similar to Sinon, with a few differences in design, philosophy, and features that could make it useful in many cases. you can read about it here, here and here.

Wallaby - Integrated Continuous Test Runner for JavaScript

Wallaby is another tool worth mentioning. It is not free, but many users recommend buying it. It runs on your IDE (it supports all major ones) and runs relevant to your code changes tests and indicates if anything fails in real time just alongside your code.

Choose Your Framework

The first choice you should probably make is what framework do you want to use and libraries to support it. It is recommended to use the tools your framework provides until a need for unique tools arises. Then it should not be hard to change to or add them.

* In short, if you want to “just get started” or looking for a fast framework for large projects, go with Jest.

* If you want a very flexible and extendable configuration, go with Mocha.
If you are looking for simplicity go with Ava.

* If you want to be really low-level, go with tape.

Here is a list of the most prominent tools with some of their pros and cons:

jasmine/jasmine

Jasmine is a testing framework providing with everything you are expected to need for your tests: a running environment, structure, reporting, assertion, and mocking tools.

• Globals- Creates test globals by default so there is no need to require them:

// "describe" is in the global scope already
// so no these require lines are not required:
//
// const jasmine = require('jasmine')
// const describe = jasmine.describe
describe('calculator', function() {
  ...
})

• Ready-To-Go- Comes with assertions, spies, mocks that are equivalent to libraries that do the same like Sinon. Libraries still can easily be used in case you need some unique features.
• Angular- Has widespread Angular support.

mochajs/mocha

Mocha is currently the most used library. Unlike Jasmine, it is used with third party assertion, mocking, and spying tools (usually Enzyme and Chai).

This means Mocha is a little harder to set up and divided into more libraries but it is more flexible and open to extensions.

For example, if you want special assertion logic, you can fork Chai and replace only Chai with your own assertion library. This can also be done in Jasmine but Mocka is more flexible in this sense.

• Community- Has many plugins and extension to test unique scenarios.
• Extensibility- Plugins, extensions and libraries such as Sinon includes features Jasmine does not have.
• Globals- Creates test structure globals by default, but obviously not assertions, spies and mocks like Jasmine- some people are surprised by this seemingly inconsistency of globals.

Jest 🃏 Painless JavaScript Testing

Jest is the testing framework recommended by Facebook. It wraps Jasmine and adds features on top of it so everything mentioned about Jasmine applies to it as well.

After reading an inhuman amount of articles and blog posts, it’s incredible how by the end of 2016 people are impressed by Jest’s speed and convenience.

• Performance- First of all Jest is considered to be faster for big projects with many test files by implementing a clever parallel testing mechanism (For example by us from our experience and in these blog posts: here, here, here, here).
• UI- Clear and convenient.
• Snapshot testing- jest-snapshot is developed and maintained by Facebook, although it can be used in almost any other framework as part of the framework’s integration of the tool or by using the right plugins.
• Improved modules mocking- Jest lets you mock heavy libraries in a very easy way to improve testing speed.
• Code coverage- Includes a powerful and fast built-in code coverage tool that is based on Istanbul.
• Support- Jest is in motion and improves in big steps as for the end of 2016 and at the first months of 2017.
• Development- jest only updates the files updated so tests are running very fast in watch mode.

avajs/ava

Ava is a minimalistic testing library that runs tests in parallel.

• Globals- Does not create any test globals thus you have more control over your tests.
• Simplicity- simple structure and assertions without a complex API while supporting many advanced features.
• Development- Ava only updates the files updated so tests are running fast in watch mode.
• Snapshot testing is supported by using jest-snapshot under the hood.

substack/tape

Tape is the simple of them all. It’s just a JS file you run with node with a very short and “to-the-point” API.

• Simplicity- Minimalistic structure and assertions without a complex API. Even more than Ava.
• Globals- Does not create any test globals thus you have more control over your tests.
• No Shared State between tests- Tape discourages the use of functions like “beforeEach” to ensure test modularity and maximum user control over the tests cycle.
• No CLI is needed- Tape is simply run anywhere JS can be run.

Unit Tests

Cover everything. Use a coverage tool like Istanbul to make sure every module in your system is covered.

Since these tests are testing separate modules, it is prefered to run them in NodeJS and not in a browser (like karma does) because running JS in browsers is slower then in NodeJS.

Integration Tests

Integration Tests- Create a list of important internal flows with empty tests as TODOs as you develop or afterward and implement these tests one by one. Consider adding UI mocking and snapshots.

Snapshot tests could be a good substitution for the traditional UI integration tests. Instead of testing parts of the UI after certain processes, you could snapshot parts of the application instead.

Consider using JSDom or Karma to run your tests in real browser.

Functional Tests

The number of permanent tools for this purpose is somehow limited, and their implementation differs very much from each other, so it is strongly suggested to try implementing some of them before taking a decision.

* In short, if you want to “just get started” with the most simple set-up, and if you want to test many environments easily, go with TestCafe.

* If you want to go with the flow and have maximum community support, if you need to be able to write tests not only in JS, Selenium is the way to go.

* If your application has no complex user interaction and graphics, for example, if you want to test a system full of forms and navigations, headless browser tools like Casper will provide you with the fastest tests.

SeleniumHQ/selenium

SeleniumHQ, better known as just Selenium, automates the browser to simulate user behavior. It is not written specifically for tests and can control a browser for many purposes by exposing a server that simulates user behavior on a browser using an API.

Selenium can be controlled in many ways and using a variety programming languages, and with some tools even without any real programming.

To our needs, however, Selenium server is controlled by a Selenium WebDriver that serves as a communication layer between our NodeJS and the server that operates the browser.

Node.js <=> WebDriver <=> Selenium Server <=> FF/Chrome/IE/Safari

The WebDriver can be imported into your testing framework and tests can be written as part of it:

describe('login form', () => {
  before(() => {
    return driver.navigate().to('http://path.to.test.app/')
  })
  it('autocompletes the name field', () => {
    driver.findElement(By.css('.autocomplete'))
      .sendKeys('John')
    driver.wait(until.elementLocated(By.css('.suggestion')))
    driver.findElement(By.css('.suggestion')).click()
    return driver.findElement(By.css('.autocomplete'))
      .getAttribute('value')
      .then(inputValue => {
        expect(inputValue).to.equal('John Doe')
      })
  })

  after(() => {
    return driver.quit()
  })
})

The WebDriver itself might be sufficient for you and indeed some people suggest using it as it is but various libraries were created to extend it wether by forking and altering it or by wrapping it.

And indeed wrapping the WebDriver might add redundant code and could make debugging harder and forking it might diverge it from the very active (as for 2017) ongoing development of it.

Still some people prefer to not use it directly. Let’s look at some of libraries for selenium operating:

angular/protractor

Protractor is a library that wraps Selenium to add it an improved syntax and special in-built hooks for Angular.

• Angular- Has special hooks, although can be successfully used with other JS frameworks too.
• Error reporting- Good mechanism.
• Mobile- No support to automate mobile Apps.
• Support- TypeScript support is available and the library is operated and maintained by the huge Angular team.

WebdriverIO — Selenium 2.0 bindings for NodeJS

WebdriverIO has it’s own implementation of the selenium WebDriver.

• Syntax- very easy and readable.
• Flexible- A very simple and agnostic from even being used for tests, flexible and extensible library.
• Community- It has good support and enthusiastic developer community that makes it reach with plugins and extensions.

Nightwatch.js

Nightwatch has it’s own implementation of the selenium WebDriver. And provides it’s own testing framework with a test server, assertions, and tools.

• Framework- Can be used with other frameworks too, but can be especially useful in case you want to run functional tests not as part of other framework.
• Syntax- looks the easiest and the most readable.
• Support- No typescript support and in general, this library seems to be slightly less supported than the others.

casperjs/casperjs

Casper is written on top of Phantom and Slimer (The same as Phantom but in FireFox’s Gecko) to provide navigation, scripting and testing utilities and abstracts away a lot of the complicated, asynchronous stuff when creating Phantom and Slimer scripts.

Casper and other headless browsers provide us with a faster but less stable way to run functional tests on UI-less browsers.

Automated browser testing for the modern web development stack | TestCafe

TestCafe is a great alternative to Selenium-Based tools.

In October 2016, the core library of TestCafe was released as an open-source JavaScript framework. There is still a paid version that offers non-JS tools like a test recorder and a customer support.

This is important because many outdated articles state that it’s code is closed and take it for a disadvantage.

It injects itself into the browser’s environment as JS scripts instead of attaching to browsers using plugins like selenium does. This allows TestCafe to run on any browser, including on mobile devices. On Selenium you have to install special plugins for every device and browser.

TestCafe is newer and more JS and test oriented. It features a very useful error reporting system that indicated the line of a failure, a very useful selectors system, and some more very useful features.

cucumber/cucumber-js

Cucumber is another useful framework for functional testing. It is a framework that arranges the previously mentioned automated tests in a slightly different way.

Cucumber help with writing tests in BDD by dividing them between the business crew that writes acceptance criteria using the Gherkin syntax and the programmers that write their tests according to it. Tests can be written in a variety of languages that are supported by the framework, including JS that we are focusing on:

features/like-article.feature: (The gherkin syntax)

Feature: A reader can share an article to social networks
  As a reader
  I want to share articles
  So that I can notify my friends about an article I liked
  Scenario: An article was opened
    Given I'm inside an article
    When I share the article
    Then the article should change to a "shared" state

features/stepdefinitions/like-article.steps.js

module.exports = function() {
  this.Given(/^I'm inside an article$/, function(callback) {
    // functional testing tool code
  })
  this.When(/^I share the article$/, function(callback) {
    // functional testing tool code
  })
    
  this.Then(/^the article should change to a "shared" state$/, function(callback) {     
    // functional testing tool code
  }) 
}

If you think this kind of arrangement can help different crews in your company to collaborate, you can find this tool useful.

Contribute

If you have anything you want me to add or change in this very short guide, just tell me. I’ll be glad to do it. I want this guide to be as accurate, as complete and as useful as possible.

Conclusion

In this very short guide, we saw the most trending testing strategies and tools in the JS community and hopefully made it easier for you to test your application.

Remember that there are many more useful tools and strategies that this guide has not covered. Some has to do with the tools that are covered here and some are completely different.

In the end, the best decisions regarding application architecture today are made by understanding general solution patterns that are developed by the very active community, and combining them with your own experience and understanding of the characteristics of your application and it’s special needs.

Oh, and writing, and rewriting, and rewriting, and rewriting, and testing different solutions :)

Happy testings :)

Thanks :)

Suggested Articles

General

• The Top 8 Essential JavaScript Automation Frameworks (June 2016)
• JavaScript Testing: Unit vs Functional vs Integration Tests (April 2016)
• What are Unit Testing, Integration Testing and Functional Testing? (April 2015)

testdouble, Sinon

• Bye-Bye, Sinon — Hello, testdouble (March 2016)
• JavaScript Testing Tool Showdown: Sinon.js vs testdouble.js (April 2017)
• testdouble.js vs. sinon.js (March 2016)

Unexpected.js

• enzyme vs unexpected-react (April 2017)

Testing Frameworks Comparison

• Testing React Applications with Karma, Jest or Mocha (August 2016)
• Let’s Compare Javascript Testing Framework (August 2016)
• Jasmine vs. Mocha, Chai, and Sinon (Updated February 2016)
• Jasmine vs. Mocha- JavaScript Testing Framework Comparison
  (January 2017)
• How to Snapshot Test Everything in Your Redux App With Jest
  (December 2016)

Jest

• Migrating to Jest on the P2P team at PayPal (November 2016)
• Jest- From 0 to 100% Coverage Real Quick (January 2017)
• The Hidden Power of Jest Matchers (January 2017)
• Testing React components with Jest and Enzyme (December 2016)
• Testing a React-Redux app using Jest and Enzyme (January 2017)
• Snapshot Testing React Components with Jest (January 2017)
• Testing an AngularJS app with Jest (October 2016)

Ava

• Effortless unit testing with AVA (April 2016)
• Javascript Testing with AVA (February 2017)
• Unit testing front-end JavaScript with AVA and jsdom (November 2016)
• Testing the Bejeezus out of React Native Apps with AVA (June 2016)

Tape

• Why I use Tape Instead of Mocha & So Should You (July 2015)
• Learn Tape

Selenium

• Protractor vs WebdriverIO vs NightWatch (November 2016)
• End to End (e2e) Testing React Apps With Selenium WebDriver And Node.js is Easier Than You Think (April 2016)
• WebdriverIO- Using JavaScript to Drive Selenium Tests (August 2016)
• JavaScript Functional Testing with Nightwatch.js (August 2016)
• Node.js End-to-End Testing with Nightwatch.js (March 2017)
• Nightmare of End-to-End Testing (September 2016)
• Protractor vs WebdriverIO vs NightWatch (November 2016)
• Stack Overflow- What is the difference between nightwatch.js And webdriver.io? (March 2016)
• Comparison of JavaScript browser automation and test specification libraries (December 2015)
• What is PhantomJS and How is it Used? (June 2016)

TestCafe

• Which automated testing tool to pick — Testcafe or Selenium WebDriver? (January 2016)
• Introducing TestCafe — Open Source Testing Framework (Octobeer 2016)
• Why not use Selenium? (October 2016)
• Functional Testing With TestCafe (March 2017)