The State of Browser Test Automation
End to End testing used to follow a very conservative approach. Everyone was using Java/Python/Ruby with Selenium and it really felt that there is no good alternative to this setup. Selenium was everywhere and at some point, it even transformed its protocol to W3C standard. Ok, we have a W3C standard now, so why do we need alternatives?
- Chrome Dev Tools Protocol — a take from Chrome Development team on bringing their own (non-W3C standard) protocol for controlling browser.
- All browser commands must run asynchronously (as they always need some time to finish)
- Each test is a list of those commands
- Each test should be linear and predictable
To sum up, we need to run all the commands one by one. Otherwise, we can make tests unpredictable. For instance, if we send 2 commands to the browser: “check element” and “click” we will have no idea which commands finish first and these tests will fail for random reasons all the time. To reduce this flakiness test automation engineers should always keep in mind that all their commands to browser must be synchronized in order to make tests pass. That’s something that doesn’t exist in Java.
Tools for Browser Testing
And now it is time to overview all testing tools over there. Also, I will share their strengths and weaknesses and implementation concerns.
Cypress.io is great! However, before choosing Cypress.io, consider its limitations. Because Cypress.io operates internally within a browser context it doesn’t have access to external features like:
- using iframes
- managing tabs
- managing multiple windows
- uploading files
- downloading files
- sending native keyboard and mouse events
With Cypress.io it is also hard to test interaction with 3rd party websites
and it has a lack of cross-browser support.
Cypress.io works great when it comes to testing single-page applications or components. Cypress.io is loved by developers! But again, the technology behind Cypress.io may not always fulfill the requirements.
When it comes to cross-browser testing, test engineers still prefer WebDriver. Yes, WebDriver is a standard and even the most sophisticated browsers, like Safari or IE are following this standard.
- Protractor — which is built on top of the official Selenium library and tries to bring Java-like syntax into JS.
As of the beginning of 2020 Protractor is the most popular WebDriver solution, however, it has a lack of maintenance at this moment. I have deep concerns about its future so I wrote a post about it.
webdriverio has a good API, great documentation, cross-browser support, native mobile apps testing support and a rich set of tools built on top of WebDriver protocol.
The technology behind WebDriver has its issues:
- hardened setup — Java, Selenium Server and ChromeDriver (or IEDriver, GeckoDriver) needed. Not as easy to start comparing to Cypress.io
- no access to browser internals — no request mocking, network interception, access to headers, or downloads out of box.
- speed — the way WebDriver operates makes it about 3 times slower comparing to all alternatives.
- strictness — WebDriver is a standard, so it is very strict about what you can do inside a browser. If an element is on a page but a center of element is not visible — you are not able to click that element. If element is moving — you can’t control it.
WebDriver is a great and stable solution with the biggest community. For classical stable setup, WebDriver is still the best on the market. Also, it has rich ecosystem and services like SauceLabs or BrowserStack which help scaling WebDriver tests.
Puppeteer is like a mix of the best from two worlds — WebDriver and Cypress.io.
Puppeteer is built on top of Chrome DevTools protocol by Chromium developers. So it is using all the API that browser can expose and that’s why it is much more powerful than classical WebDriver. If WebDriver is limited to its standard (all methods should behave similarly in all browsers), Puppeteer is focused around Chrome and Firefox only, so it can drive innovations much faster.
Unlike Cypress.io, Puppeteer gives you true external control of a browser. You can manage tabs, iframes, incognito sessions, network (mocking requests), file uploads, downloads. Almost everything! But its speed is comparable to Cypress.io, yes Puppeteer is really fast.
To be correct I need to list the weaknesses of Puppeteer. But I don’t see any technical issues with it. It gives everything possible to you so you could write the most sophisticated browser tests. It is easy to install, has nice API, and very flexible in control. It’s a very straightforward tool, so unlike Cypress, it doesn’t give you nice features like automatic-waiting for elements, or waiting for animations. If you need those features — you need to develop them by yourself.
We didn’t hear any notable activity from the Puppeteer team for some time. Puppeteer 2.0 was released without any major changes and no real compatibility breaks. It seemed that Puppeteer's development entered the boring stage when only new minor improvements and bugfixes were introduced. We didn’t know that the team behind Puppeteer was working on another project which takes the best from Puppeteer but extends it to Firefox and Safari.
That’s right, officially announced on January 2020 Playwright already looks as a gamechanger. It has polished API from Puppeteer with all its features. Under the hood, Playwright patches Firefox and Webkit (Safari engine) to include its automation API. So this brings a very low-level and very fast protocol for browser control. This approach won’t work with any closed source browser. However, with Microsoft finally switching to Chromium we won’t need any other desktop browser to support in tests.
Playwright team also included several enhancements comparing to Puppeteer:
- reduce the need for timeouts
- better detection for element visibility before interaction
I didn’t have a chance to try it (yet) but looking forward to doing so. I expect Playwright to be highly loved by developers and test automation engineers. The idea and technology look great, as well as the team behind it.
Another great technology with its own pros and cons. TestCafe does a very nice trick — instead of controlling browser on its own, it starts a proxy server that intercepts any calls in a browser and injects their script inside of them.
This is how it works: when you access
google.com in a test, the browser is actually opening:
The real address is a proxy server that sends a request to google.com but appends its own script into every HTML received. This server also starts and closes the browser, however, it doesn’t control a browser when a test when it is running. So, as in Cypress.io, your tests actually work inside a browser (internally).
This solution makes TestCafe the fastest tool with cross-browser support on the market. Indeed, if you don’t need to control a browser, you can use any browser you want! Even Safari, even IE, even mobile Chrome! Every browser can be taken by TestCafe and this will be much faster than WebDriver (because tests are executed inside the browser). The issues with this technology are mostly edge cases which is hard to be listed. Basically, TestCafe should work for the majority of tests, but when it doesn’t work, you won’t have a clear idea why. And no debug tools to help you with that.
TestCafe has a rich set of features, it is fast and has nice community around it. You should definitely take a look at it, if cross-browser support is your primary requirement.
Cypress.io, WebDriver, Puppeteer, TestCafe — they are all great. They solve the problem of testing in their own way. However, all those frameworks have their own API and their own syntax. Once we started writing tests we won’t be able to migrate to another engine when we hit an unsolvable problem.
- What if we started writing tests in Puppeteer but then we received a requirement to support Safari in tests?
- We started with WebDriver but we need more browser control that only Puppeteer can give us?
- We started in TestCafe but some tests are unstable
- We started with Cypress.io but our website contains iframes or we need to run tests in two windows.
So here is the idea behind CodeceptJS — a meta-framework for browser test automation. CodeceptJS takes all popular browser driving engines, wraps them and provides you with a unified syntax. Unlike other tools listed in this article, CodeceptJS doesn’t control the browser on its own, it delegates the control to WebDriver, Puppeteer, or TestCafe. This allows CodeceptJS to be focused more on developers' experience. For instance, it brings page objects, step retries, and interactive pause out of the box. CodeceptJS is as simple in usage as Cypress.io but allows you to use different technologies.