E2E testing framework round-up

As developers, we program computers to do repetitive tasks so humans don’t have to. Sanity-checking the flows of an application after it has been updated is a classic example of a repetitive task which should be automated. However poor tooling has historically made writing and maintaining these end-to-end tests painfully inefficient, to the point that many teams don’t automate. In starting a new project with a strong focus on automation I spent some (often frustrating) time evaluating the popular E2E testing tools — Protractor, TestCafe, Cypress, and Puppeteer.

Requirements

1. E2E tests should be as realistic as possible, so the tests should issue simple instructions that map directly to user actions in the browser, and the test runner should follow these instructions exactly.
2. The tests need to run as part of the automated build process in an environment that is as close as possible to the local environment
3. Handling of asynchronicity should be robust and easy to reason about

These tests are time consuming to maintain, and inter-browser variability is lower now than in the past, so I’m comfortable only running the tests against Chrome.

Methodology

I wrote a suite of tests for each of the tools, covering:

• Logging in
• Resetting the environment
• Testing a normal user happy path
• Testing an administrator happy path

And added this suite to the Gitlab build pipeline.

Protractor

I’ve used Protractor a lot, and have found that it works fine for basic scenarios, but becomes flaky with complexity:

1. I’ve found that Protractor’s ‘waiting for Angular’ functionality ties tests into what is happening behind the scenes, which fundamentally fails point 1 above and for me is a classic example of trying to be too fancy and ending up with flakiness. The test runner does things — especially around loading pages — which are not written in the tests.
2. The most popular way to run Protractor tests in the cloud is to use Sauce labs. While Sauce labs provides some great tooling, after a few hours of debugging tests that passed when I pointed my local Protractor instance at the live server, but failed on Sauce labs, I gave up trying to fill the gaps.
3. Protractor does a lot of fancy stuff with the control flow so that you can write asynchronous tests that look like synchronous tests. This is great when it works, but is very difficult to reason about when it goes wrong. I’m aware that you can use explicit promises / async, but that goes against the examples in the style guide.

For me Protractor fails by trying so hard to be simple that it ends up, in real world use cases, being too complicated.

TestCafe

TestCafe has some great tooling and runs from its own server (ie ‘noSelenium’), which means local tests execute in an environment that is very close to the build process environment.

1. TestCafe also waits for requests to complete, and will wait for an element to be visible before trying to click on it (rather than failing, as Protractor would do). There is a ‘roles’ feature for saving the state of different users. However with both these features I found the execution was unpredictable, and there were actions during test execution (for example page reloads) which were not written in the tests. The way roles manages local storage didn’t work in some flows, which took a lot of time to debug.
2. TestCafe was relatively straightforward to set up on Gitlab, and the execution matched what I was seeing locally. The ability to store videos and screenshots as Gitlab artifacts is a great help for debugging.
3. TestCafe uses chained promises for control flow, and I found these to be easy to read and robust.

Cypress

Cypress is also a ‘noSelenium’ option, with great developer-orientated tooling.

The issue with Cypress was that it was tricky to set up on Gitlab, and threw a lot of errors about cross-site security which I wasn’t seeing locally. After doing some research I decided that, despite the excellent local developer experience, running on Gitlab’s looked too complicated.

Puppeteer

Puppeteer is a tool for controlling Chrome, rather than a test runner — but you can use any normal JS test runner to drive it (I ended up using Jasmine to be consistent with my unit tests).

1. By default Puppeteer doesn’t do any waiting or state management. This means that tests take more code (with explicit waits for elements to be present), but execute is a way that is extremely close to the way a user interacts with the application.
2. Setting up Puppeteer on Gitlab was easy, and the behavior is very close to how the tests run locally. I’ve now implemented monitoring by running Puppeteer in an AWS lambda on a schedule — this ability to run in different environments without unpleasant surprises is a big win.
3. The most popular pattern for Puppeteer seems to be explicitly using async / await. Again this results in a bit more code, but is very easy to reason about and transparent.

Decision

The tooling for JS / Typescript development nowadays is awesome, and it was disappointing to see that E2E testing frameworks are so poor — largely because they try to be too simple and end up being too complex. Puppeteer is the only one of these four where I felt the test runner was doing exactly and only what was written in the tests, and consequently that I would be able to debug any issues without having to go down an abstraction layer to deal with flow control, Angular state, storage management, etc.

I’ve been running Puppeteer now for about a month now and it has been consistently reliable and stable.