An Overview of JavaScript Testing in 2019

This guide is intended to catch you up with the most important reasoning, terms, tools, and approaches to JavaScript testing in 2019. It combines information from the best recent articles (linked at the bottom) and adds from our own experience at Welldone Software Solutions, where we have implemented different testing solutions for different products over many years.

It’s a little long. I know :D :D. But whoever reads and understands this guide, can safely assume they know the big picture of the state of JavaScript testing in the web development community for 2019. This is a great reason to share it with all your co-workers, family, and friends :)

Look at the slogan of Cypress.io above. They are right. The web has evolved and it was amazing to follow the evolution of website testing in recent years.

Just a couple of years ago website testings was slow, expensive, hard to implement, not deterministic, and in general not too fun to work on.

But “There’s never been a better time to be a JavaScript developer than now” according to The State of JavaScript 2018. This is especially true in regards to JavaScript Testing, with a marked increase of “overall happiness” from 3.2/5 to 3.8/5 in the last year.

Today’s cutting edge website testing tools are fast, informative, easy to work with, and they make your testing and developing experience much more enjoyable.

Looking forward, I forecast a large entrance of AI into the field of automated testing. Some such tools already exist, and actively improving the workflow and experience of thousands of developers.
  • If you see anything I missed or got wrong, leave a comment and I’ll address it right away.
  • I’ll keep this guide updated throughout 2019, so if you see anything outdated, let me know by commenting below.
  • Notice the links at the bottom of the page. Reading them will take you from just understanding the big picture, to becoming an expert (in theory).

Enjoy :)

Test Types

You can read about different test types in more depth here and here and here.

In general, the most important test types for a website are:

  • Unit Tests- Testing of individual units like functions or classes by supplying input and making sure the output is as expected:
  • Integration Tests- Testing processes across several units to achieve their goals, including their side effects:
  • Functional Tests (also known as e2e tests)- Testing how scenarios function on the product itself, by controlling the browser or the website. This tests usually ignore the internal structure of the application entirety and looks at them from the outside like on a black box.

Running Tests

Tests can run in the browser by creating an HTML page with the test libraries and tests included as JS files.

Tests can also be executed in Node.js by simply importing the tests and dependent libraries. jsdom is commonly used in Node.js to simulate a browser-like environment using pure JavaScript. It provides window, document, body, location, cookies, selectors and whatever you get when you run your JS inside the browser, but it renders nothing real. This is different from a “headless mode” because for a headless mode you need a real browser and it can even take screenshots, unlike jsdom.

We suggest the second method (Node.js + jsdom) because it’s much faster than running tests in the browser. The first method may be more reliable because you’re using the the same exact software which will render your site in real life. However, using jsdom is lighter and should be sufficient for most use cases.

Test Tools Types

Test tools can be divided into the following functionalities. Some provide us with only one functionality, and some provide us with a combination.

To achieve the most flexible set functionality, it’s common to use a combination of several tools.

  1. Test launchers launch your tests in the browser or Node.js with user config using the CLI or UI. This can also be achieved by opening the browser manually. (Karma, Jasmine, Jest, TestCafe, Cypress)
  2. Testing structure providers help you arrange your test files. (Mocha, Jasmine, Jest, Cucumber, TestCafe, Cypress)
  3. Assertion functions check if the results a test returns are as expected. (Chai, Jasmine, Jest, Unexpected, TestCafe, Cypress)
  4. Generate and display test progress and results. (Mocha, Jasmine, Jest, Karma, TestCafe, Cypress)
  5. Mocks, spies, and stubs to isolate certain parts of tests and catch their side effects. (Sinon, Jasmine, enzyme, Jest, testdouble)
  6. Generate and compare snapshots of component and data structures to make sure changes from previous runs are intended. (Jest, Ava)
  7. Generate code coverage reports to know how much of the code is covered by your tests. (Istanbul, Jest, Blanket)
  8. Browser Controllers simulate user actions for Functional Tests. (Nightwatch, Nightmare, Phantom, Puppeteer, TestCafe, Cypress)
  9. Visual Regression Tools are used to compare your site to its previous versions visually by using image comparison techniques.
    (Applitools, Percy, Wraith, WebdriverCSS)

Let’s explain some of the terms mentioned above:

Test launchers are given a list of tests, and the various configuration and scaffolding necessary to run those tests (what browsers to run in, what babel plugins to use, how to format output, etc).

Testing structure refers to the organization of your tests. Nowadays, tests are usually organized in a BDD structure that supports behavior-driven development (BDD). It often looks like this:

Assertion functions are used to make sure that tested variables contain the expected value. They usually look like this, the first two styles being the most common:

TIP: Here is a nice article about advanced Jasmine and Jest assertions.

Spies provide us with information about functions. For example, how many times were they called, in which cases, and by whom?

Spies are used in integration tests to make sure that the side effects of a process are as expected. For example, how many times was a calculation function called during some process?

Notice how we run father.execute() and count how many times father.child ran during this process.

Stubbing or dubbing replaces selected methods of existing modules with user-supplied functions in order to ensure expected behavior during the test.

To ensure user.isValid() will always return true during a test, where you test a different component, you can use:

Mocks or Fakes are used to fake certain modules or behaviors to test different parts of a processes.

Sinon can, for example, fake a server to ensure offline, fast and expected responses when testing a certain flow.

Snapshot Testing is when you compare a data structure to an expected one.

The following example, from the official Jest documentation, shows a snapshot test of a certain Link component.

It doesn’t actually renders and takes a picture of the component, but it saves its internal data in a separate file like this:

When the test runs, and the new snapshot differs from the last one, the developer is prompted to confirm that the change is intended.

Notice: Snapshots are usually made to compare component representation data but they can also compare other types of data, like redux stores or the inner structure of different units in the application.

Browsers can be controlled by drivers that are installed on top of them and control the browser using different methods. This is how selenium works:

Node.js <=> WebDriver <=> FF/Chrome/IE/Safari drivers <=> browser

Or by script injection of a JS code that has access to the whole application environment: DOM, network, cookies etc… and can raise events that simulate user behavior. For example:
document.getElementByID('someButton').dispatchEvent(clickEvent).

Node.js <=> FF/Chrome/IE/Safari injected script <=> Simulated events

Putting it All Together

Start by choosing the testing structure and syntax (2) you like, assertion functions (3) library, and decide how do you want to run the tests (1).

Some frameworks like Jest, Jasmine, TestCafe, and Cypress provide all of these out of the box. Some of them provide only some of the functionality and a combination of libraries should be used: (mocha + chai + sinon)

We also suggest creating two different processes. One for running unit and integration tests and another one for Functional Tests. This is because functional tests usually take longer, especially when running the test suite on several different browsers.

Think about when it’s appropriate to run each type of test. For example:

Unit + integration on every change, functional tests only before commits.

Unit Tests

Should cover all small pure units of the application- utils, services and helpers. Provide all these units with simple and edge case inputs and make sure their outputs are as expected using the assertion functions (3).
Also make sure to use a code coverage reporting tool (7) to know which units are covered.

Unit tests are one of the reasons to use functional programming and pure functions as much as possible.
The purer your application is, the easier you can test it.

Integration Tests

Old school tests were focused on unit testing and resulted in applications where many small parts were working but the processes as a whole kept on failing.
Integration tests, on the other hand, detect cases where a unit is refactored and passes its tests but a process that depends on it fails.

The best demonstration of why testing more than each part of a system separately is important can be seen in this great GIF.

I created this GIF based on a slide in this great lecture by
Viv Richards that I recommend to watch.

It is also important to remember that in the real world, for the reasons of imperfect design and the widespread use of black boxes, not all units are pure and not all units are testable- some units can be tested only as part of a bigger process.

Integration tests should cover important cross-module processes. Sometimes they extend to processes across several classes and sometimes to testing different systems like Front-End-Back-End interactions.

Comparing to unit tests, you would probably benefit from using spies (5) to ensure expected side effects instead of just asserting the output and stubs (5) to mock and modify the parts of the process that are not in the specific test.

Also, as opposed to unit tests, a browser or browser-like environment (jsdom) could be required to access the window.

Component snapshot tests (4) fall into this category as well. They provide us with a way to test how processes affect selected components without actually rendering them into a browser or browser-like environment.

Functional Tests

Sometimes the quick and effective unit and integration tests are not enough.

Functional tests control browsers (7) and simulate user behavior on these environments (clicking, typing, scrolling etc…) and make sure these scenarios actually work from the point of view of an end user.

Also it is worth noting that many services provide you with devices and browsers to run these tests on. Here are even more services like these.

Visual regression testing tools (8) can also be set up to verify that the different screens of your applications are ok visually by a smart comparison of screenshots. These screenshots are usually taken as part of your Functional Tests or by executing a separate session of browser automation.

Applitools smart visual regression tools

List of General Prominent Testing Tools

There are dozens of great tools out there. I couldn’t include all of them here but I tried to include the most important to know, best maintained , and the most adopted tools in the following list:

jsdom

jsdom is a JavaScript implementation of the WHATWG DOM and HTML standards. In other words, jsdom simulates a browser’s environment without running anything but plain JS.

As mentioned before, in this simulated browser environment, tests can run really fast. The drawback of jsdom is that not everything can be simulated outside a real browser (you can’t take a screenshot for example) so using it will limit your test’s reach.

It is worth mentioning that the JS community rapidly improves jsdom and the current version is very close to a real browser.

Electron

The Electron framework lets you write cross-platform desktop applications using JavaScript, HTML and CSS. It also has a headless mode.

It has a huge community and many important applications are built on top of it, so it is supposed to stay up to date:
Atom, Slack, Skype, GitHub Desktop and many more.

Testing tools like Cypress.io use Electron to launch tests with maximum control of the browser.

Istanbul

Istanbul will tell you how much of your code is covered with unit tests. It will report on statement, line, function and branch coverage in percentages so you will understand better what is left to cover.

Karma

Karma hosts a test server with a special web page to run your tests in the page’s environment. This page can be run across many browsers and browser-like environments including jsdom.

Chai

Chai is the most popular assertion library. It has many plugins and extensions.

Unexpected

Unexpected is an assertion library with a slightly different syntax from Chai. It is also extensible so assertions can be more advanced with libraries that are based on it like unexpected-react that you can read about more in depth here.

Sinon.JS

Sinon has very powerful standalone test spies, stubs and mocks for JavaScript that works with any unit testing framework.

testdouble.js

testdouble is a less popular library that does what Sinon does, and claims to do it better, with a few differences in design, philosophy, and features that could make it useful in many cases. You can read about it here, here and here.

Wallaby

Wallaby is another tool worth mentioning. It is not free, but many users recommend buying it. It runs on your IDE (it supports all major ones) and runs tests that are relevant to your code changes and indicates if anything fails in real time alongside your code.

Cucumber

Cucumber help with writing tests in BDD by dividing them between the acceptance criteria files using the Gherkin syntax and the tests that correspond to them.

Tests can be written in a variety of languages that are supported by the framework, including JS, which we are focusing on:

Many teams will find this syntax more convenient than TDD.

Choose Your Unit and Integration Tests Framework

The first choice you should probably make is which framework you want to use. It is recommended to use the tools your framework provides until a need for unique tools arises.

* In short, if you want to “just get started” or are looking for a fast framework for large projects, you can’t go wrong with Jest.
* If you want a very flexible and extendable configuration, go with Mocha.
* If you are looking for simplicity go with Ava.
* If you want to be really low-level, go with tape.

Here is a list of the most prominent tools with some of their characteristics:

Jest

Jest is the testing framework created and maintained by Facebook. It spiked in popularity and became the most popular library throughout 2017 (Judging by State of Javascript 2017 vs 2018)

It is based on Jasmine which we will discuss later. Over time, Facebook replaced most of its functionality and added a lot of features on top of it.

After reading a huge amount of articles and blog posts, it’s incredible how people are impressed by Jest’s speed and convenience.
  • Performance- First of all Jest is considered to be faster for big projects with many test files by implementing a clever parallel testing mechanism, ( This is true in our experience and also discussed in these blog posts: here, here, here, here, here).
  • UI- Clear and convenient.
  • Ready-To-Go- Comes with assertions, spies, and mocks that are equivalent to libraries that do the same like Sinon. Libraries still can easily be used in case you need some unique features.
  • Globals- Like in Jasmine, it creates test globals by default so there is no need to require them. This can be considered bad since it makes your tests less flexible and less controllable, but in most cases it just makes your life easier:
  • Snapshot testing- jest-snapshot is developed and maintained by Facebook, although it can be used in almost any other framework as part of the framework’s integration of the tool or by using the right plugins.
  • Improved modules mocking- Jest provides you with an easy way to mock heavy modules to improve testing speed. For example a service can be mocked to resolve a promise instead of making a network request.
  • Code coverage- Includes a powerful and fast built-in code coverage tool that is based on Istanbul.
  • Reliability- Although this is a relatively young library, throughout 2017 and 2018, Jest stabilized and is now considered reliable. It is currently supported by all the major IDEs and tools.
  • Development- jest only updates the files updated, so tests are running very fast in watch mode.

jasmine

Jasmine is the testing framework that Jest is based on. Why would you prefer Jasmine over Jest? It has been around for longer and it has a huge amount of articles, tools, and questions answered in various forums that were all created by the community over many years.

Also, Angular still suggests using it over Jest, although Jest is perfectly suitable to run Angular tests as well, and many people do it.

  • Ready-To-Go- Comes with everything you need to start testing.
  • Globals- Comes with all the important testing features in the global scope as well.
  • Community- It has been on the market since 2009 and gathered a vast amount of articles, suggestions and tools that are based on it.
  • Angular- Has widespread Angular support for all its versions and it is recommended in the official Angular documentation.

mocha

Mocha is the most used library. Unlike Jasmine, it is used with third party assertions, mocking, and spying tools (usually Sinon and Chai).

This means Mocha is a little harder to set up and divided into more libraries but it is more flexible and open to extensions.

For example, if you want special assertion logic, you can fork Chai and replace only Chai with your own assertion library. This can also be done in Jasmine but in Mocha this change will be more clear and obvious.

  • Community- Has many plugins and extension to test unique scenarios.
  • Extensibility- Very extensible, to the point where plugins, extensions and libraries are designed only to run on top of it.
  • Globals- Creates test structure globals by default, but obviously not assertions, spies and mocks like Jasmine. some people are surprised by this seemingly inconsistency of globals.

AVA

Ava is a minimalistic testing library that runs tests in parallel.

  • Ready-To-Go- Comes with everything you need to start testing (besides spying and dubbing that you can add easily). Uses the following syntax for test structure and assertions, and runs in Node.js:
  • Globals- As seen above, it does not create any test globals, so you have more control over your tests.
  • Simplicity- Simple structure and assertions without a complex API while supporting many advanced features.
  • Development- Ava only updates the files updated so tests will run fast in watch mode.
  • Speed- Runs tests in parallel as separate Node.js processes.
  • Snapshot testing is supported as part of the framework.

tape

Tape is the simplest of them all. It’s just a JS file you run with node with a very short and “to-the-point” API.

  • Simplicity- Minimalistic structure and assertions without a complex API. Even more than Ava.
  • Globals- Does not create any test globals so you have more control over your tests.
  • No Shared State between tests- Tape discourages the use of functions like “beforeEach” to ensure test modularity and maximum user control over the testsing cycle.
  • No CLI is needed- Tape will simply run anywhere JS can run.

Functional Testing Tools

First of all, as mentioned above, here and here you can find great articles about service providers that will host the machines where tests run and help you run these tests on different devices and browsers.

The tools for the purpose of functional testing differ very much from each other in their implementation, philosophy, and API, so it is strongly suggested to invest time in understanding the different solutions and testing them on your product.

* In short, if you want to “just get started” with a simple to set-up cross-browser all-in-one tool, go with TestCafe.
* For a convenient UI, clear documentation, cool tools and overall fun all-in-one tool Functional Testing experience go with Cypress.io.
* If you prefer older and more time-proven tools, you can “just get started” with Nightwatch.js.
* If you prefer older and more time-proven tools, with the maximum community support and flexability, WebdriverIO is the way to go.
* If you want the most reliable and Angular friendly solution, use Protractor.

selenium

Selenium and tools that rely on it dominated the market of Functional Tests for years. It is not written specifically for testing and can control a browser for many purposes by exposing a driver that controls browsers using add-ins and browser extensions.

Node.js <=> WebDriver <=> FF/Chrome/IE/Safari drivers <=> browser

Selenium WebDriver can be accessed in many different ways and using a variety of programming languages, and with some tools even without any real programming.

The WebDriver can be imported into your testing framework and tests can be written as part of it:

The WebDriver itself might be sufficient for you and indeed some people suggest using it as is, but various libraries were created to extend it either by forking and altering it or by wrapping it.

And indeed wrapping the WebDriver might add redundant code and could make debugging harder, whereas forking it might diverge it from the very active ongoing development of the WebDriver.

Still, some people prefer to not use it directly. Let’s look at some of the libraries for selenium:

Protractor

Protractor is a library that wraps Selenium and provides us with improved syntax and special built-in hooks for Angular.

  • Angular- Has special hooks, although it can successfully be used with other JS frameworks too. Angular official documentation suggests using this tool.
  • Error reporting- Good mechanism.
  • Support- TypeScript support is available and the library is operated and maintained by the huge Angular team.

WebdriverIO

WebdriverIO has its own implementation of the selenium WebDriver.

  • Syntax- very easy and readable.
  • Flexible- A very simple, flexible, and extensible library.
  • Community- It has good support and enthusiastic developer community.

Nightwatch

Nightwatch has its own implementation of the selenium WebDriver. And provides its own testing framework with a test server, assertions, and tools.

  • Framework- Can be used with other frameworks too, but can be especially useful in case you want to run functional tests not as part of another framework.
  • Syntax- looks the easiest and the most readable.
  • Support- No typescript support and in general, this library seems to be slightly less supported than the others.

Appium

Apium provides an API similar to Selenium for testing websites on a mobile device using the following tools:

So if you use Selenium or Selenium based tools, you can also use Apium to test on mobile devices.

TestCafe

TestCafe is a great alternative to Selenium-Based tools. It was rewritten and open-sourced at the end of 2016.

TestCafe had also a paid version that offered non-programming testing tools. It has been deprecated and will be replaced by the new TestCafe Studio, which is free of charge at the moment but will become a commercial product when it is officially released in a couple of months.

TestCafe injects itself into the website as JavaScript scripts instead of controlling the browsers themselves like Selenium does. This allows it to run on any browser, including on mobile devices, and have full control over the JavaScript execution loop.

Cypress

Cypress is a direct competitor of TestCafe. They are doing relatively the same, which is injecting tests into a website, but they try to do it in a more modern, flexible and convenient way.

The difference between them is that Cypress.io runs itself in the browser and controls your tests from there where TestCafe runs in Node.js and controls the tests through a serialized communication with its injected script in the browser.

The library is relatively new (moved from closed beta to public beta at October 2017) but they already have many adopters and enthusiasts.

  • Parallel testing was introduced in version 3.10.
  • Documentation- Solid and clear.
  • Native access to all your application’s variables without serialization (TestCafe on the other hand turns objects to JSON, sends them to Node.js as text and then parses them back to objects).
  • Very convenient running and debugging tools- Easy debugging and logging of the test process.
  • No cross-browser Support- Only support chrome right now, and not headless. Runs in Electron in headless mode. They are working on it as this article is being created. (And we will update the article once they do it)
  • Some use-cases are missing but in constant development such as lack of HTML5 drag-n-drop.
  • Using Mocha as the test structure provider makes its use pretty standard and allow your functional tests to be built in the same structure as the rest of your tests.

Puppeteer

Puppeteer is a Node.js library, developed by Google. It provides a convenient Node.js API to control Chrome or Headless Chrome.

Headless Chrome is just a regular Chrome v59+ that is launched with the
--headless flag. When Chrome is run in headless mode, it exposes an API to control it, and as said before, Puppeteer is the JavaScript tool that Google provides to control it.

Here it is worth mentioning that Firefox has also released their headless mode at the end of 2017.

Notice that different testing tools can also use Headless Chrome and Firefox. For example: TestCafe, Karma, Cypress.

  • Puppeteer is relatively new, but it has a big community that uses and develops tools and wrappers around it.
  • Since it is native and uses the latest Chrome engine, it is very fast.
  • One major drawback of Headless Chrome (thus of Puppeteer as well) is that it doesn’t supports extensions like Flash and probably won’t in the near future.

PhantomJS

Phantom implements the chromium engine to create a controllable Chrome-like headless browser. It was a great tool to run in headless mode until Google announcement of “Puppeteer”.

It’s main maintainer, Vitaliy Slobodin, no longer works on it, and its development was suspended and its repository archived.

Nightmare

Nightmare is a Functional Testing library that offers a very simple test syntax.

It uses Electron which uses Chromium to control the browser’s behavior.

Doesn’t seem to be maintained recently. Probably because of the introduction of “Puppeteer” as well which provides you with the same features out of the box.

CodeceptJS

Like CucumberJS which was discussed above, Codecept provides another abstraction over different libraries’ API’s to make your interactions with tests use a slightly different philosophy that focuses on user behavior.

Here is how it looks:

And here is the list of libraries that can be executed using this code. All discussed above.

WebDriverIO, Protractor, Nightmare, Appium, Puppeteer.

If you believe this syntax is better for your needs, give it a shot.

Visual Regression Testing

The visual regression testing tools are consists roughly of the following:

  • Techniques and integrations to automate the browser or to run as part of Functional Testing tools discussed above, including in the CLI for CI.
  • Taking smart screenshots as images and as DOM snapshots.
  • Images and DOM comparison techniques to spot differences sometimes even using advanced AI.
  • UI for humans to approve, reject and improve the comparison mechanism to only show what’s relevant for the user.

There are a lot of tools of this type in the market, but it feels like this field still has a long way to do.

Also I noticed that paid tools in the visual regression testing category are much better than the free ones.

Applitools

  • Easy to set up.
  • Uses AI to make the comparison technique and human input regarding differences and negation of false positives very powerful.
  • It can be integrated conveniently with many of the tools discussed above.
  • Has a free and paid flexible plans, including special pricing for startup companies and non-profits.

Percy

  • Easy to set up.
  • Uses smart comparison techniques.
  • Convenient human input regarding differences.
  • It can be integrated conveniently with many of the tools discussed above.
  • It has great integrations with great tools.
  • Has free and paid flexible plans.

Happo

Happo is a paid visual regression testing tool. It hooks into the CI to compare the visual appearance of UI components before and after a change.

Screenshots can be taken in different browsers and across different screen sizes to ensure consistent cross-browser and responsive styling of your application.

Paid with a free plan for open source projects.

Gemini

An open source utility by Yandex.

It uses LooksSame (by Yandex as well) which can be used separately to achieve similar goals.

It is much more simple and limited than the paid tools mentioned above, but for simple websites it can be enough.

BackstopJS

An open source visual regression utility that runs on Chrome Headless with Puppeteer and CI support.

AyeSpy

An open source utility by the Times Tooling team at News UK.

Uses selenium docker to create visual regression tests on Chrome / Firefox.

reg-suit

An open source library that compares images, generates reports and saves them on the cloud. Very convenient if you want to add visual regression tests to an existing functional test. Simply add steps of taking screenshots to your existing test flow and use it to compare these screenshots.

Differencify

Another open source Chrome Headless using Puppeteer testing tool with nice integrations with Jest snapshots. Can run in docker and generates convenient reports.

No Coding Functional Testing Tools

testim

Opens your application in a separate window and uses a browser extension to record your manual interactions with the application as test scenarios.

Uses machine learning to help you record and validate test scenarios. Cross Browser and has some nice integrations with many CI and Collaboration tools.

Has free and paid flexible plans.

Screener

Lets you record your tests using a chrome extension, has in depth visual regression reporting. Has some nice integrations like with storybook different CI tools and BrowserStack and Sauce Labs.

Not free.

Other tools of this type (you are free to suggest more in the comments):

Conclusion

We reviewed the most trending testing strategies and tools in the web development community and hopefully made it easier for you to test your sites.

In the end, the best decisions regarding application architecture today are made by understanding general patterns that are trending in the very active community of developers, and combining them with your own experience and the characteristics of your application.

Oh, and writing, and rewriting, and rewriting, and rewriting, and testing different solutions :)

Happy testings :)

Thanks :)

Suggested Links