JavaScript: the Future of End to End Testing
UPDATE: A few days after this post was published a new browser automation tool emerged! Playwright provides native cross-browser support without using WebDriver protocol. I added it to this post. And I must say, that JavaScript indeed is the future and the future happens today!
The State of Browser Test Automation
End to End testing used to follow a very conservative approach. Everyone was using Java/Python/Ruby with Selenium and it really felt that there is no good alternative to this setup. Selenium was everywhere and at some point, it even transformed its protocol to W3C standard. Ok, we have a W3C standard now, so why do we need alternatives?
We always need alternatives and the tech community is great for creating ones! However, before the magnificent rise of JavaScript, no other platforms did provide any sane and popular solution. Nowadays, in the JavaScript ecosystem, we already have 3 alternatives to Selenium and a room is open for new ones! So, in my opinion, switching all browser test automation to JavaScript is a matter of time now. This is a very ambitious statement, but from business, perspective choosing faster and more stable tools over old Selenium in Java should dramatically increase the productivity and outcome for software developers in the test.
The only pain point of switching is JavaScript. Most test automation engineers are trained to work with Selenium in classical backend languages like Java and Python, and switching to a new language or tool can be a problem for them. Test automation was done in Java, all new engineers were trained from scratch for Java and Selenium, and this kept repeating, year by year.
Javascript vs Java vs Python
Ok, but what made JavaScript this popular and why it is winning the battle? At this moment, it’s hard to say it is winning in numbers but it wins in technology. Let me list a few points on what driven JavaScript browser testing evolution:
- JavaScript is already inside the browser. Obvious point. Why to learn two languages if you can use one?
- Electron — a framework that allowed to wrap a browser inside JavaScript engine. It allowed creating a set of Selenium alternatives like Nightmare.js and Cypress.io
- Chrome Dev Tools Protocol — a take from Chrome Development team on bringing their own (non-W3C standard) protocol for controlling browser.
I can also mention the simplicity of a language, it’s flexibility, npm, and the rise of frontend applications as additional points that boosted interest to browser testing in JS. However, unlike traditional setups (Java, Python) JavaScript was never designed for end to end testing.
That’s right.
The problem here is the major feature of JavaScript — asynchrony is absolutely useless, and even dangerous when it comes to testing in a browser.
- All browser commands must run asynchronously (as they always need some time to finish)
- Each test is a list of those commands
- Each test should be linear and predictable
To sum up, we need to run all the commands one by one. Otherwise, we can make tests unpredictable. For instance, if we send 2 commands to the browser: “check element” and “click” we will have no idea which commands finish first and these tests will fail for random reasons all the time. To reduce this flakiness test automation engineers should always keep in mind that all their commands to browser must be synchronized in order to make tests pass. That’s something that doesn’t exist in Java.
Tools for Browser Testing
And now it is time to overview all testing tools over there. Also, I will share their strengths and weaknesses and implementation concerns.
Cypress.io
The most popular Selenium alternative nowadays is brought by a Cypress.io company. Cypress.io was built with the idea that if JavaScript is running inside a browser, we should execute our test within it as well. They built a wrapper on top of a browser using Electron and packed it as a feature full standalone solution. Cypress.io got popular as it was very simple in use, especially for frontend engineers. Working with Cypress.io is as simple as using JQuery in good old days. Cypress.io is faster than Selenium, it is more stable, and can take a benefit from running within an application.
Cypress.io is great! However, before choosing Cypress.io, consider its limitations. Because Cypress.io operates internally within a browser context it doesn’t have access to external features like:
- using iframes
- managing tabs
- managing multiple windows
- uploading files
- downloading files
- sending native keyboard and mouse events
With Cypress.io it is also hard to test interaction with 3rd party websites
and it has a lack of cross-browser support.
Cypress.io works great when it comes to testing single-page applications or components. Cypress.io is loved by developers! But again, the technology behind Cypress.io may not always fulfill the requirements.
WebDriver
When it comes to cross-browser testing, test engineers still prefer WebDriver. Yes, WebDriver is a standard and even the most sophisticated browsers, like Safari or IE are following this standard.
Talking of WebDriver implementations I need to say in JavaScript we have two popular implementations:
- Protractor — which is built on top of the official Selenium library and tries to bring Java-like syntax into JS.
- webdriverio — alternative JavaScript implementation.
As of the beginning of 2020 Protractor is the most popular WebDriver solution, however, it has a lack of maintenance at this moment. I have deep concerns about its future so I wrote a post about it.
webdriverio has a good API, great documentation, cross-browser support, native mobile apps testing support and a rich set of tools built on top of WebDriver protocol.
The technology behind WebDriver has its issues:
- hardened setup — Java, Selenium Server and ChromeDriver (or IEDriver, GeckoDriver) needed. Not as easy to start comparing to Cypress.io
- no access to browser internals — no request mocking, network interception, access to headers, or downloads out of box.
- speed — the way WebDriver operates makes it about 3 times slower comparing to all alternatives.
- strictness — WebDriver is a standard, so it is very strict about what you can do inside a browser. If an element is on a page but a center of element is not visible — you are not able to click that element. If element is moving — you can’t control it.
WebDriver is a great and stable solution with the biggest community. For classical stable setup, WebDriver is still the best on the market. Also, it has rich ecosystem and services like SauceLabs or BrowserStack which help scaling WebDriver tests.
Puppeteer
Puppeteer is like a mix of the best from two worlds — WebDriver and Cypress.io.
Puppeteer is built on top of Chrome DevTools protocol by Chromium developers. So it is using all the API that browser can expose and that’s why it is much more powerful than classical WebDriver. If WebDriver is limited to its standard (all methods should behave similarly in all browsers), Puppeteer is focused around Chrome and Firefox only, so it can drive innovations much faster.
Unlike Cypress.io, Puppeteer gives you true external control of a browser. You can manage tabs, iframes, incognito sessions, network (mocking requests), file uploads, downloads. Almost everything! But its speed is comparable to Cypress.io, yes Puppeteer is really fast.
The only problem with Puppeteer — it is a general-purpose library, and not a test automation solution. So developers usually pair it with Jest or CodeceptJS.
To be correct I need to list the weaknesses of Puppeteer. But I don’t see any technical issues with it. It gives everything possible to you so you could write the most sophisticated browser tests. It is easy to install, has nice API, and very flexible in control. It’s a very straightforward tool, so unlike Cypress, it doesn’t give you nice features like automatic-waiting for elements, or waiting for animations. If you need those features — you need to develop them by yourself.
Puppeteer is great as a technology but has a higher learning curve than Cypress.io and less learning materials than WebDriver. It requires experienced JavaScript engineers who can set it up and build a testing framework around it.
Playwright
We didn’t hear any notable activity from the Puppeteer team for some time. Puppeteer 2.0 was released without any major changes and no real compatibility breaks. It seemed that Puppeteer's development entered the boring stage when only new minor improvements and bugfixes were introduced. We didn’t know that the team behind Puppeteer was working on another project which takes the best from Puppeteer but extends it to Firefox and Safari.
That’s right, officially announced on January 2020 Playwright already looks as a gamechanger. It has polished API from Puppeteer with all its features. Under the hood, Playwright patches Firefox and Webkit (Safari engine) to include its automation API. So this brings a very low-level and very fast protocol for browser control. This approach won’t work with any closed source browser. However, with Microsoft finally switching to Chromium we won’t need any other desktop browser to support in tests.
Playwright team also included several enhancements comparing to Puppeteer:
- reduce the need for timeouts
- better detection for element visibility before interaction
I didn’t have a chance to try it (yet) but looking forward to doing so. I expect Playwright to be highly loved by developers and test automation engineers. The idea and technology look great, as well as the team behind it.
TestCafe
Another great technology with its own pros and cons. TestCafe does a very nice trick — instead of controlling browser on its own, it starts a proxy server that intercepts any calls in a browser and injects their script inside of them.
This is how it works: when you access google.com in a test, the browser is actually opening:
http://192.168.0.110:38475/xVPmF1Kwb/http://google.com/The real address is a proxy server that sends a request to google.com but appends its own script into every HTML received. This server also starts and closes the browser, however, it doesn’t control a browser when a test when it is running. So, as in Cypress.io, your tests actually work inside a browser (internally).
This solution makes TestCafe the fastest tool with cross-browser support on the market. Indeed, if you don’t need to control a browser, you can use any browser you want! Even Safari, even IE, even mobile Chrome! Every browser can be taken by TestCafe and this will be much faster than WebDriver (because tests are executed inside the browser). The issues with this technology are mostly edge cases which is hard to be listed. Basically, TestCafe should work for the majority of tests, but when it doesn’t work, you won’t have a clear idea why. And no debug tools to help you with that.
TestCafe has a rich set of features, it is fast and has nice community around it. You should definitely take a look at it, if cross-browser support is your primary requirement.
CodeceptJS
Cypress.io, WebDriver, Puppeteer, TestCafe — they are all great. They solve the problem of testing in their own way. However, all those frameworks have their own API and their own syntax. Once we started writing tests we won’t be able to migrate to another engine when we hit an unsolvable problem.
- What if we started writing tests in Puppeteer but then we received a requirement to support Safari in tests?
- We started with WebDriver but we need more browser control that only Puppeteer can give us?
- We started in TestCafe but some tests are unstable
- We started with Cypress.io but our website contains iframes or we need to run tests in two windows.
So here is the idea behind CodeceptJS — a meta-framework for browser test automation. CodeceptJS takes all popular browser driving engines, wraps them and provides you with a unified syntax. Unlike other tools listed in this article, CodeceptJS doesn’t control the browser on its own, it delegates the control to WebDriver, Puppeteer, or TestCafe. This allows CodeceptJS to be focused more on developers' experience. For instance, it brings page objects, step retries, and interactive pause out of the box. CodeceptJS is as simple in usage as Cypress.io but allows you to use different technologies.
While Cypress.io is good for frontend engineers and matches their experience, CodeceptJS focuses more on pure end to end testing. So the test in CodeceptJS looks more like a test plan and not the JavaScript code. So even manual QAs can start working with it learning JavaScript step by step.
Conclusion
I believe this year will be game-changing for test automation. Test automation market is going to hit 30 billion dollars in next years. And JavaScript will drive this growth and the innovation in test automation. Consider switching your browser tests to JavaScript today — but choose the tool wisely, before picking up the most popular Cypress.io or Protractor estimate the alternatives like CodeceptJS, webdriver.io or TestCafe. They can match your requirements better.
