Evaluating Cypress and TestCafe for end to end testing
Two frameworks have appeared on the scene recently eschewing the long held belief that end to end testing of web applications means building upon Selenium.
Cypress and TestCafe are similar in many ways but have some important differences.
Let’s look at some similarities
- Both are open source (MIT licensed) end to end testing frameworks written in JavaScript.
- Both run partially in the browser and partially in Node.JS.
- Both transparently retry assertions which eliminates a lot of the flakiness associated with Selenium based tests.
- Both are under heavy development with very responsive developers and similar sized communities.
And now some differences
Cypress uses (a fork of) Mocha as its test runner with Chai for assertions and Sinon for mocking. This means most JavaScript developers will feel immediately at home. It also does some promise queueing magic to make the asynchronous code look more synchronous.
TestCafe uses its own test runner which is initially a bit strange, especially as it applies its terminology in an unusual way like referring to a group of tests as a fixture. But I prefer that it makes you explicitly use promises and async/await to manage execution.
It’s a shame that neither expose an API that would let you use the same test runner you already use for unit and integration tests.
TestCafe works by serving the test site via a proxy server. The server injects scripts into the page which can inspect and control elements on the page. It does some clever stuff to even allow interacting with native alerts, file upload inputs and iframes. This means it works in any web browser, including mobile devices and cloud services like BrowserStack and SauceLabs.
Cypress works by controlling the browser via its proprietary automation APIs. This means it needs a new driver for every supported browser. As of early 2018 it only supports Chrome and family (Chromium, Electron) though Firefox is in the works. There are plans to support Edge and Safari and possibly older browsers by falling back to Selenium WebDriver. The CEO of Cypress makes the case that cross browser testing may not be as important as you think but some users are seeing inconsistencies even between Electron and Chrome.
Cypress runs your actual test code in the browser process whereas TestCafe runs it in Node. This means Cypress tests have access to real DOM elements but in TestCafe communication between your tests and the DOM must be serialised.
Since TestCafe tests run in Node, you can call out to parts of your Node server application directly from the tests. This is useful for setting up and clearing database fixtures or even starting and stopping the test server. In Cypress you’re limited to communicating with your app via HTTP or executing shell commands.
TestCafe uses standard CSS selectors to locate elements but Cypress uses jQuery selectors which have some useful extra capabilities like :parent and :first. On the other hand TestCafe has framework specific extensions for React, Angular, Vue and Aurelia that let you use component names as selectors.
TestCafe has nice console output for test failures (as seen in some other test runners like Jest) that shows which assertion failed, in context. This largely mitigates the issues of having many assertions in a single test.
Cypress goes one further by having a dedicated Electron app that shows your tests side by side with the site under test. Hovering over a test step shows a snapshot of the DOM for that step. And the killer feature is that it watches your code and re-runs the tests when you change anything. This makes for an awesome developer experience. It’s almost — almost — actually fun to write tests.
As I’ve been writing this post, the TestCafe developers have launched a similar feature called TestCafe Live. Which brings me to my next point.
As I mentioned, both frameworks are in very active development. That means it’s a great time to involved and help shape the product. But it also means there are still bugs and missing features.
For example Cypress doesn’t natively support file uploads. You can do it but it’s fiddly and requires creating a File object and triggering an event in whatever shape your app expects. Native file upload support in Cypress is planned.
Meanwhile I’ve encountered several issues trying to input text into contenteditable elements with TestCafe.
So which is best
We ended up going with TestCafe because of its better browser support. Our current project, building an open source journal submission platform for Cambridge-based eLife Sciences, needs to be tested in a wide range of browsers including Internet Explorer. If you’re only supporting evergreen browsers then this may not matter.
For many use cases there’s really not much between them. They’re both a huge improvement over the previous generation of tools and I look forward to seeing where they go next.
Written by Tamlyn Rhodes — Full stack web developer for YLD.
