Look at the slogan of Cypress.io above. They are right. The web has evolved and it was amazing to follow the evolution of website testing in recent years.
Just a couple of years ago website testings was slow, expensive, hard to implement, not deterministic, and in general not too fun to work on.
Today’s cutting edge website testing tools are fast, informative, easy to work with, and they make your testing and developing experience much more enjoyable.
Looking forward, I forecast a large entrance of AI into the field of automated testing. Some such tools already exist, and actively improving the workflow and experience of thousands of developers.
- If you see anything I missed or got wrong, leave a comment and I’ll address it right away.
- I’ll keep this guide updated throughout 2019, so if you see anything outdated, let me know by commenting below.
- Notice the links at the bottom of the page. Reading them will take you from just understanding the big picture, to becoming an expert (in theory).
In general, the most important test types for a website are:
- Unit Tests- Testing of individual units like functions or classes by supplying input and making sure the output is as expected:
- Integration Tests- Testing processes across several units to achieve their goals, including their side effects:
- Functional Tests (also known as e2e tests)- Testing how scenarios function on the product itself, by controlling the browser or the website. This tests usually ignore the internal structure of the application entirety and looks at them from the outside like on a black box.
Tests can run in the browser by creating an HTML page with the test libraries and tests included as JS files.
We suggest the second method (Node.js + jsdom) because it’s much faster than running tests in the browser. The first method may be more reliable because you’re using the the same exact software which will render your site in real life. However, using jsdom is lighter and should be sufficient for most use cases.
Test Tools Types
Test tools can be divided into the following functionalities. Some provide us with only one functionality, and some provide us with a combination.
To achieve the most flexible set functionality, it’s common to use a combination of several tools.
- Test launchers launch your tests in the browser or Node.js with user config using the CLI or UI. This can also be achieved by opening the browser manually. (Karma, Jasmine, Jest, TestCafe, Cypress)
- Testing structure providers help you arrange your test files. (Mocha, Jasmine, Jest, Cucumber, TestCafe, Cypress)
- Assertion functions check if the results a test returns are as expected. (Chai, Jasmine, Jest, Unexpected, TestCafe, Cypress)
- Generate and display test progress and results. (Mocha, Jasmine, Jest, Karma, TestCafe, Cypress)
- Mocks, spies, and stubs to isolate certain parts of tests and catch their side effects. (Sinon, Jasmine, enzyme, Jest, testdouble)
- Generate and compare snapshots of component and data structures to make sure changes from previous runs are intended. (Jest, Ava)
- Generate code coverage reports to know how much of the code is covered by your tests. (Istanbul, Jest, Blanket)
- Browser Controllers simulate user actions for Functional Tests. (Nightwatch, Nightmare, Phantom, Puppeteer, TestCafe, Cypress)
- Visual Regression Tools are used to compare your site to its previous versions visually by using image comparison techniques.
(Applitools, Percy, Wraith, WebdriverCSS)
Let’s explain some of the terms mentioned above:
Test launchers are given a list of tests, and the various configuration and scaffolding necessary to run those tests (what browsers to run in, what babel plugins to use, how to format output, etc).
Testing structure refers to the organization of your tests. Nowadays, tests are usually organized in a BDD structure that supports behavior-driven development (BDD). It often looks like this:
Assertion functions are used to make sure that tested variables contain the expected value. They usually look like this, the first two styles being the most common:
TIP: Here is a nice article about advanced Jasmine and Jest assertions.
Spies provide us with information about functions. For example, how many times were they called, in which cases, and by whom?
Spies are used in integration tests to make sure that the side effects of a process are as expected. For example, how many times was a calculation function called during some process?
Notice how we run
father.execute() and count how many times
father.child ran during this process.
Stubbing or dubbing replaces selected methods of existing modules with user-supplied functions in order to ensure expected behavior during the test.
user.isValid() will always return
true during a test, where you test a different component, you can use:
Mocks or Fakes are used to fake certain modules or behaviors to test different parts of a processes.
Sinon can, for example, fake a server to ensure offline, fast and expected responses when testing a certain flow.
Snapshot Testing is when you compare a data structure to an expected one.
The following example, from the official Jest documentation, shows a snapshot test of a certain
It doesn’t actually renders and takes a picture of the component, but it saves its internal data in a separate file like this:
When the test runs, and the new snapshot differs from the last one, the developer is prompted to confirm that the change is intended.
Notice: Snapshots are usually made to compare component representation data but they can also compare other types of data, like redux stores or the inner structure of different units in the application.
Node.js <=> WebDriver <=> FF/Chrome/IE/Safari drivers <=> browser
Or by script injection of a JS code that has access to the whole application environment: DOM, network, cookies etc… and can raise events that simulate user behavior. For example:
Node.js <=> FF/Chrome/IE/Safari injected script <=> Simulated events
Putting it All Together
Start by choosing the testing structure and syntax (2) you like, assertion functions (3) library, and decide how do you want to run the tests (1).
Some frameworks like Jest, Jasmine, TestCafe, and Cypress provide all of these out of the box. Some of them provide only some of the functionality and a combination of libraries should be used: (mocha + chai + sinon)
We also suggest creating two different processes. One for running unit and integration tests and another one for Functional Tests. This is because functional tests usually take longer, especially when running the test suite on several different browsers.
Think about when it’s appropriate to run each type of test. For example:
Unit + integration on every change, functional tests only before commits.
Should cover all small pure units of the application- utils, services and helpers. Provide all these units with simple and edge case inputs and make sure their outputs are as expected using the assertion functions (3).
Also make sure to use a code coverage reporting tool (7) to know which units are covered.
Unit tests are one of the reasons to use functional programming and pure functions as much as possible.
The purer your application is, the easier you can test it.
Old school tests were focused on unit testing and resulted in applications where many small parts were working but the processes as a whole kept on failing.
Integration tests, on the other hand, detect cases where a unit is refactored and passes its tests but a process that depends on it fails.
The best demonstration of why testing more than each part of a system separately is important can be seen in this great GIF.
It is also important to remember that in the real world, for the reasons of imperfect design and the widespread use of black boxes, not all units are pure and not all units are testable- some units can be tested only as part of a bigger process.
Integration tests should cover important cross-module processes. Sometimes they extend to processes across several classes and sometimes to testing different systems like Front-End-Back-End interactions.
Comparing to unit tests, you would probably benefit from using spies (5) to ensure expected side effects instead of just asserting the output and stubs (5) to mock and modify the parts of the process that are not in the specific test.
Also, as opposed to unit tests, a browser or browser-like environment (jsdom) could be required to access the
Component snapshot tests (4) fall into this category as well. They provide us with a way to test how processes affect selected components without actually rendering them into a browser or browser-like environment.
Sometimes the quick and effective unit and integration tests are not enough.
Functional tests control browsers (7) and simulate user behavior on these environments (clicking, typing, scrolling etc…) and make sure these scenarios actually work from the point of view of an end user.
Visual regression testing tools (8) can also be set up to verify that the different screens of your applications are ok visually by a smart comparison of screenshots. These screenshots are usually taken as part of your Functional Tests or by executing a separate session of browser automation.
List of General Prominent Testing Tools
There are dozens of great tools out there. I couldn’t include all of them here but I tried to include the most important to know, best maintained , and the most adopted tools in the following list:
As mentioned before, in this simulated browser environment, tests can run really fast. The drawback of jsdom is that not everything can be simulated outside a real browser (you can’t take a screenshot for example) so using it will limit your test’s reach.
It is worth mentioning that the JS community rapidly improves jsdom and the current version is very close to a real browser.
Testing tools like Cypress.io use Electron to launch tests with maximum control of the browser.
Istanbul will tell you how much of your code is covered with unit tests. It will report on statement, line, function and branch coverage in percentages so you will understand better what is left to cover.
The main goal for Karma is to bring a productive testing environment to developers. The environment being one where…karma-runner.github.io
Karma hosts a test server with a special web page to run your tests in the page’s environment. This page can be run across many browsers and browser-like environments including jsdom.
chai — BDD / TDD assertion framework for node.js and the browser that can be paired with any testing framework.github.com
Chai is the most popular assertion library. It has many plugins and extensions.
Unexpected is an assertion library with a slightly different syntax from Chai. It is also extensible so assertions can be more advanced with libraries that are based on it like unexpected-react that you can read about more in depth here.
testdouble is a less popular library that does what Sinon does, and claims to do it better, with a few differences in design, philosophy, and features that could make it useful in many cases. You can read about it here, here and here.
Wallaby is another tool worth mentioning. It is not free, but many users recommend buying it. It runs on your IDE (it supports all major ones) and runs tests that are relevant to your code changes and indicates if anything fails in real time alongside your code.
Cucumber help with writing tests in BDD by dividing them between the acceptance criteria files using the Gherkin syntax and the tests that correspond to them.
Tests can be written in a variety of languages that are supported by the framework, including JS, which we are focusing on:
Many teams will find this syntax more convenient than TDD.
Choose Your Unit and Integration Tests Framework
The first choice you should probably make is which framework you want to use. It is recommended to use the tools your framework provides until a need for unique tools arises.
* In short, if you want to “just get started” or are looking for a fast framework for large projects, you can’t go wrong with Jest.
* If you want a very flexible and extendable configuration, go with Mocha.
* If you are looking for simplicity go with Ava.
* If you want to be really low-level, go with tape.
Here is a list of the most prominent tools with some of their characteristics:
It is based on Jasmine which we will discuss later. Over time, Facebook replaced most of its functionality and added a lot of features on top of it.
After reading a huge amount of articles and blog posts, it’s incredible how people are impressed by Jest’s speed and convenience.
- Performance- First of all Jest is considered to be faster for big projects with many test files by implementing a clever parallel testing mechanism, ( This is true in our experience and also discussed in these blog posts: here, here, here, here, here).
- UI- Clear and convenient.
- Ready-To-Go- Comes with assertions, spies, and mocks that are equivalent to libraries that do the same like Sinon. Libraries still can easily be used in case you need some unique features.
- Globals- Like in Jasmine, it creates test globals by default so there is no need to require them. This can be considered bad since it makes your tests less flexible and less controllable, but in most cases it just makes your life easier:
- Snapshot testing- jest-snapshot is developed and maintained by Facebook, although it can be used in almost any other framework as part of the framework’s integration of the tool or by using the right plugins.
- Improved modules mocking- Jest provides you with an easy way to mock heavy modules to improve testing speed. For example a service can be mocked to resolve a promise instead of making a network request.
- Code coverage- Includes a powerful and fast built-in code coverage tool that is based on Istanbul.
- Reliability- Although this is a relatively young library, throughout 2017 and 2018, Jest stabilized and is now considered reliable. It is currently supported by all the major IDEs and tools.
- Development- jest only updates the files updated, so tests are running very fast in watch mode.
Jasmine is the testing framework that Jest is based on. Why would you prefer Jasmine over Jest? It has been around for longer and it has a huge amount of articles, tools, and questions answered in various forums that were all created by the community over many years.
Also, Angular still suggests using it over Jest, although Jest is perfectly suitable to run Angular tests as well, and many people do it.
- Ready-To-Go- Comes with everything you need to start testing.
- Globals- Comes with all the important testing features in the global scope as well.
- Community- It has been on the market since 2009 and gathered a vast amount of articles, suggestions and tools that are based on it.
- Angular- Has widespread Angular support for all its versions and it is recommended in the official Angular documentation.
This means Mocha is a little harder to set up and divided into more libraries but it is more flexible and open to extensions.
For example, if you want special assertion logic, you can fork Chai and replace only Chai with your own assertion library. This can also be done in Jasmine but in Mocha this change will be more clear and obvious.
- Community- Has many plugins and extension to test unique scenarios.
- Extensibility- Very extensible, to the point where plugins, extensions and libraries are designed only to run on top of it.
- Globals- Creates test structure globals by default, but obviously not assertions, spies and mocks like Jasmine. some people are surprised by this seemingly inconsistency of globals.
Ava is a minimalistic testing library that runs tests in parallel.
- Ready-To-Go- Comes with everything you need to start testing (besides spying and dubbing that you can add easily). Uses the following syntax for test structure and assertions, and runs in Node.js:
- Globals- As seen above, it does not create any test globals, so you have more control over your tests.
- Simplicity- Simple structure and assertions without a complex API while supporting many advanced features.
- Development- Ava only updates the files updated so tests will run fast in watch mode.
- Speed- Runs tests in parallel as separate Node.js processes.
- Snapshot testing is supported as part of the framework.
Tape is the simplest of them all. It’s just a JS file you run with node with a very short and “to-the-point” API.
- Simplicity- Minimalistic structure and assertions without a complex API. Even more than Ava.
- Globals- Does not create any test globals so you have more control over your tests.
- No Shared State between tests- Tape discourages the use of functions like “beforeEach” to ensure test modularity and maximum user control over the testsing cycle.
- No CLI is needed- Tape will simply run anywhere JS can run.
Functional Testing Tools
First of all, as mentioned above, here and here you can find great articles about service providers that will host the machines where tests run and help you run these tests on different devices and browsers.
The tools for the purpose of functional testing differ very much from each other in their implementation, philosophy, and API, so it is strongly suggested to invest time in understanding the different solutions and testing them on your product.
* In short, if you want to “just get started” with a simple to set-up cross-browser all-in-one tool, go with TestCafe.
* For a convenient UI, clear documentation, cool tools and overall fun all-in-one tool Functional Testing experience go with Cypress.io.
* If you prefer older and more time-proven tools, you can “just get started” with Nightwatch.js.
* If you prefer older and more time-proven tools, with the maximum community support and flexability, WebdriverIO is the way to go.
* If you want the most reliable and Angular friendly solution, use Protractor.
Selenium and tools that rely on it dominated the market of Functional Tests for years. It is not written specifically for testing and can control a browser for many purposes by exposing a driver that controls browsers using add-ins and browser extensions.
Node.js <=> WebDriver <=> FF/Chrome/IE/Safari drivers <=> browser
Selenium WebDriver can be accessed in many different ways and using a variety of programming languages, and with some tools even without any real programming.
The WebDriver can be imported into your testing framework and tests can be written as part of it:
The WebDriver itself might be sufficient for you and indeed some people suggest using it as is, but various libraries were created to extend it either by forking and altering it or by wrapping it.
And indeed wrapping the WebDriver might add redundant code and could make debugging harder, whereas forking it might diverge it from the very active ongoing development of the WebDriver.
Still, some people prefer to not use it directly. Let’s look at some of the libraries for selenium:
Protractor is a library that wraps Selenium and provides us with improved syntax and special built-in hooks for Angular.
- Angular- Has special hooks, although it can successfully be used with other JS frameworks too. Angular official documentation suggests using this tool.
- Error reporting- Good mechanism.
- Support- TypeScript support is available and the library is operated and maintained by the huge Angular team.
A nodejs bindings implementation for selenium 2.0/webdriver.webdriver.io
WebdriverIO has its own implementation of the selenium WebDriver.
- Syntax- very easy and readable.
- Flexible- A very simple, flexible, and extensible library.
- Community- It has good support and enthusiastic developer community.
Write efficient and straightforward end-to-end tests in Node.js which run against a Selenium/WebDriver server.nightwatchjs.org
Nightwatch has its own implementation of the selenium WebDriver. And provides its own testing framework with a test server, assertions, and tools.
- Framework- Can be used with other frameworks too, but can be especially useful in case you want to run functional tests not as part of another framework.
- Syntax- looks the easiest and the most readable.
- Support- No typescript support and in general, this library seems to be slightly less supported than the others.
Apium provides an API similar to Selenium for testing websites on a mobile device using the following tools:
- iOS 9.3+: Apple’s XCUITest
- Before iOS 9.3: Apple’s UIAutomation
- Android 4.2+: Google’s UiAutomator/UiAutomator2
- Android 2.3+: Google’s Instrumentation. (Instrumentation support is provided by bundling a separate project, Selendroid)
- Windows Phone: Microsoft’s WinAppDriver
So if you use Selenium or Selenium based tools, you can also use Apium to test on mobile devices.
TestCafe is a simple and powerful framework for testing websites and apps. It allows you to easily create, maintain and…devexpress.github.io
TestCafe is a great alternative to Selenium-Based tools. It was rewritten and open-sourced at the end of 2016.
TestCafe had also a paid version that offered non-programming testing tools. It has been deprecated and will be replaced by the new TestCafe Studio, which is free of charge at the moment but will become a commercial product when it is officially released in a couple of months.
- Fast to set up- Npm install and run your first test on any browser you want.
- Cross Browser and Devices- Supports many browsers and devices and can be used with SauceLabs or BrowserStack which provide devices and browsers for your tests. This includes running of tests in Headless Chrome and Headless Firefox which will discussed later.
- Parallel Testing- TestCafe can run your tests on several browser instances at once. This practice can significantly decrease your testing time.
- Convenient Error Reporting
- Own Ecosystem- TestCafe uses its own test structure. It could be quite convenient, especially because UI tests usually run separately from other tests, but some people dislike it.
Fast, easy and reliable testing for anything that runs in a browser. Install Cypress in seconds and take the pain out…www.cypress.io
Cypress is a direct competitor of TestCafe. They are doing relatively the same, which is injecting tests into a website, but they try to do it in a more modern, flexible and convenient way.
The difference between them is that Cypress.io runs itself in the browser and controls your tests from there where TestCafe runs in Node.js and controls the tests through a serialized communication with its injected script in the browser.
- Parallel testing was introduced in version 3.10.
- Documentation- Solid and clear.
- Native access to all your application’s variables without serialization (TestCafe on the other hand turns objects to JSON, sends them to Node.js as text and then parses them back to objects).
- Very convenient running and debugging tools- Easy debugging and logging of the test process.
- No cross-browser Support- Only support chrome right now, and not headless. Runs in Electron in headless mode. They are working on it as this article is being created. (And we will update the article once they do it)
- Some use-cases are missing but in constant development such as lack of HTML5 drag-n-drop.
- Using Mocha as the test structure provider makes its use pretty standard and allow your functional tests to be built in the same structure as the rest of your tests.
Puppeteer is a Node.js library, developed by Google. It provides a convenient Node.js API to control Chrome or Headless Chrome.
Headless Chrome is just a regular Chrome v59+ that is launched with the
Here it is worth mentioning that Firefox has also released their headless mode at the end of 2017.
Notice that different testing tools can also use Headless Chrome and Firefox. For example: TestCafe, Karma, Cypress.
- Puppeteer is relatively new, but it has a big community that uses and develops tools and wrappers around it.
- Since it is native and uses the latest Chrome engine, it is very fast.
- One major drawback of Headless Chrome (thus of Puppeteer as well) is that it doesn’t supports extensions like Flash and probably won’t in the near future.
It’s main maintainer, Vitaliy Slobodin, no longer works on it, and its development was suspended and its repository archived.
Nightmare is a Functional Testing library that offers a very simple test syntax.
It uses Electron which uses Chromium to control the browser’s behavior.
Doesn’t seem to be maintained recently. Probably because of the introduction of “Puppeteer” as well which provides you with the same features out of the box.
Like CucumberJS which was discussed above, Codecept provides another abstraction over different libraries’ API’s to make your interactions with tests use a slightly different philosophy that focuses on user behavior.
Here is how it looks:
And here is the list of libraries that can be executed using this code. All discussed above.
If you believe this syntax is better for your needs, give it a shot.
Visual Regression Testing
The visual regression testing tools are consists roughly of the following:
- Techniques and integrations to automate the browser or to run as part of Functional Testing tools discussed above, including in the CLI for CI.
- Taking smart screenshots as images and as DOM snapshots.
- Images and DOM comparison techniques to spot differences sometimes even using advanced AI.
- UI for humans to approve, reject and improve the comparison mechanism to only show what’s relevant for the user.
There are a lot of tools of this type in the market, but it feels like this field still has a long way to do.
Also I noticed that paid tools in the visual regression testing category are much better than the free ones.
Applitools is an application visual management and AI-powered visual UI testing and monitoring software. Use our visual…applitools.com
- Easy to set up.
- Uses AI to make the comparison technique and human input regarding differences and negation of false positives very powerful.
- It can be integrated conveniently with many of the tools discussed above.
- Has a free and paid flexible plans, including special pricing for startup companies and non-profits.
- Easy to set up.
- Uses smart comparison techniques.
- Convenient human input regarding differences.
- It can be integrated conveniently with many of the tools discussed above.
- It has great integrations with great tools.
- Has free and paid flexible plans.
Happo is a cross-browser screenshot testing service. Prevent visual regressions, visualize your UI changes, and keep a…happo.io
Happo is a paid visual regression testing tool. It hooks into the CI to compare the visual appearance of UI components before and after a change.
Screenshots can be taken in different browsers and across different screen sizes to ensure consistent cross-browser and responsive styling of your application.
Paid with a free plan for open source projects.
An open source utility by Yandex.
It is much more simple and limited than the paid tools mentioned above, but for simple websites it can be enough.
Catch CSS curve balls. Contribute to garris/BackstopJS development by creating an account on GitHub.github.com
An open source visual regression utility that runs on Chrome Headless with Puppeteer and CI support.
Uses selenium docker to create visual regression tests on Chrome / Firefox.
recycle: Visual Regression Testing tool. Contribute to reg-viz/reg-suit development by creating an account on GitHub.github.com
An open source library that compares images, generates reports and saves them on the cloud. Very convenient if you want to add visual regression tests to an existing functional test. Simply add steps of taking screenshots to your existing test flow and use it to compare these screenshots.
Differencify is a library for visual regression testing - NimaSoroush/differencifygithub.com
Another open source Chrome Headless using Puppeteer testing tool with nice integrations with Jest snapshots. Can run in docker and generates convenient reports.
No Coding Functional Testing Tools
Testim uses machine learning automation testing to improve the testing around your software by integrating with all…www.testim.io
Opens your application in a separate window and uses a browser extension to record your manual interactions with the application as test scenarios.
Uses machine learning to help you record and validate test scenarios. Cross Browser and has some nice integrations with many CI and Collaboration tools.
Has free and paid flexible plans.
Screener is a cloud-based automation platform for visual and functional UI testing. Test your web applications and UI…screener.io
Lets you record your tests using a chrome extension, has in depth visual regression reporting. Has some nice integrations like with storybook different CI tools and BrowserStack and Sauce Labs.
Other tools of this type (you are free to suggest more in the comments):
We reviewed the most trending testing strategies and tools in the web development community and hopefully made it easier for you to test your sites.
In the end, the best decisions regarding application architecture today are made by understanding general patterns that are trending in the very active community of developers, and combining them with your own experience and the characteristics of your application.
Oh, and writing, and rewriting, and rewriting, and rewriting, and testing different solutions :)
Happy testings :)
- Top UI Test Automation Best Practices You Should Follow (November 2017)
- Introduction to Headless Browser Testing (September 2017)
- How to set up E2E browser testing for your GitHub project (With TestCafe) (February 2017)
- Top 12 Browser Compatibility Testing Tools (June 2017)
- Top 10 Cross Browser Compatibility Testing Tools In 2019 (2019)
- Bye-Bye, Sinon — Hello, testdouble (March 2016)
- testdouble.js vs. sinon.js (March 2016)
- enzyme vs unexpected-react (April 2017)
- React Testing — Jest or Mocha? (May 2017)
- How to Snapshot Test Everything in Your Redux App With Jest
- The Hidden Power of Jest Matchers (January 2017)
- Testing React components with Jest and Enzyme (December 2016)
- Snapshot Testing React Components with Jest (January 2017)
- Testing an AngularJS app with Jest (October 2016)
- Effortless unit testing with AVA (April 2016)
- Why I use Tape Instead of Mocha & So Should You (July 2015)
- Learn Tape
- Hybrid Application Testing with Protractor and Appium (March 2017)
- Headless Chrome vs PhantomJS Benchmark (September 2017)
- Protractor vs WebdriverIO vs NightWatch (November 2016)
- End to End (e2e) Testing React Apps With Selenium WebDriver And Node.js is Easier Than You Think (April 2016)
- Node.js End-to-End Testing with Nightwatch.js (March 2017)
- Nightmare of End-to-End Testing (September 2016)
- Protractor vs WebdriverIO vs NightWatch (November 2016)
- Stack Overflow- What is the difference between nightwatch.js And webdriver.io? (March 2016)
- What is PhantomJS and How is it Used? (June 2016)
- Setting up End-to-End Testing in Angular Project with CodeceptJS (December 2017)
- Which automated testing tool to pick — Testcafe or Selenium WebDriver? (January 2016)
- Introducing TestCafe — Open Source Testing Framework (Octobeer 2016)
- Why not use Selenium? (October 2016)
- Functional Testing With TestCafe (March 2017)
- UI Test Automation Frameworks Showdown: TestCafé versus Nightwatch.js (February 2018)
- Cypress: The future of end-to-end testing for web applications (February 2018)
- Evaluating Cypress and TestCafe for end to end testing (March 2018)
- Robo-Testing your Website Without Writing Code (October 2016)