Which Automation Tool is the Best: Selenium Web Driver| Cypress | WebdriverIO | TestCafe | Playwright
Which automation tool | framework should I choose for my new project?
In my opinion, this is one of the most frequently asked & searched question, whenever a new automation tool needs to be selected for the new project. If you joined a company, where the test automation tool and programming language have already been decided, and there is no intention to change it, then you are one of the luckiest one — you can skip the reading, unless you are curious about other automation tools :)
The idea of the article is to show you different automation tools that can be used for setting up the automation framework for functional E2E tests. I will not cover standard characteristics of each tool, since it might take too long. However, I will try to cover special characteristics which give remarkable advantage to that tool, and also cover the limitations that could be the most important thing, when we are searching for a new tool. So, it is neither easy, nor logical to say one tool is the best, or better than the other one — it completely depends on your needs and product characteristics.
Overview
Cypress and TestCafe are native JavaScript test frameworks that have been in development for the past few years. Playwright is an open-source Node.js library started by Microsoft for automating browsers, which also supports multiple programming languages. They are each gaining popularity among developers as their capabilities become more robust. The advantages of these frameworks is that they are written in JavaScript, which many developers prefer, and they run within a browser, rather than remotely executing tests like Selenium.
Selenium WebDriver is on the other hand can be used together with JS, however it is widely used with Java programming language. Recently it has been popular with Python and Ruby as well. Despite all these JS based automation tools, Selenium WebDriver is still the most adopted test automation framework used today. It is also the preferred test automation framework for QA testers who run comprehensive test automation strategies. This might change in the future — we will see… In order to stay competitive, Selenium WebDriver has also made impressive improvements as they prepare for the major release for the Selenium 4.
Lastly, WebdriverIO was first introduced back in 2015. It offers a custom implementation of the Selenium WebDriver API.
Let’s have a look at their short definitions… ✍
What is Selenium? Selenium is an open-source tool that is used for automating the tests carried out on web browsers (Web applications are tested using any web browser). Selenium automates browsers. Primarily, it is for automating web applications for testing purposes, but is certainly not limited to just that. Repetitive web-based administration tasks can also be automated as well.
What is Cypress? Cypress is a front end automated testing application created for the modern web. Cypress is built on a new architecture and runs in the same run-loop as the application being tested.
What is WebdriverIO? Webdriver/Selenium 2.0 JavaScript bindings for Node.js. WebdriverIO is written in JavaScript and uses Selenium under the hood. It also comes with its own inbuilt test runner and supports other testing frameworks like Jasmine, Cucumber, and Mocha.
What is TestCafe? It is a pure node.js end-to-end solution for testing web apps. It takes care of all the stages: starting browsers, running tests, gathering test results and generating reports.
What is PlayWright? Playwright by Microsoft is the newest addition to the Headless Browser Testing frameworks. Playwright is a web test automation library that tests against the underlying engine for the most popular browsers. Playwright leverages the DevTools protocol to write powerful, stable automated tests.
It is also good to mention here that playwright is an advance version of Puppeteer. Puppeteer, an open source web automation tool built by Google. However, Puppeteer did not offer support for Safari or Firefox. Microsoft hired developers from the Puppeteer team to build Playwright as an advanced version of that tool that provided more features and broader browser support.
Special Features
Below features are naturally not limited to what in this article covered. The goal is to give an insight of each tool.
Selenium Web Driver: The new features offered in Selenium 4, including Relative Locators, which locate the elements with respect to nearby specified elements such as above(), below(), toLeftOf(), near()...etc.
With the help of newWindow() method, without having to create a new driver object to manage multiple applications in the same test, a brand new or tabbed window can be created and opened in the same session.
Chrome DevTools Protocol, which allows user to take advantage of the enhanced visibility into the browser that a debugging protocol gives, an option to grab a screenshot at UI element level, enhanced Selenium Grid deployment, observability and enhanced exceptions are the part of new features.
Cypress: A very handy feature of Cypress is the live reloading capability.
This means that as you write your testscript and hit ‘save’, the Test Runner picks up the file and reruns the test. Even if this means breaking off the already running test. This gives you almost instant feedback on the test you are writing.
There is a screenshot for every step, of your script, which can be quite helpful if there is any false passing or failing test, good for debugging as well!
Cypress waits for the elements to become visible, the animation to complete, DOM to load, the XHR and AJAX calls to be finished, etc. Hence, there is no need for implicit and explicit waits to be defined.
WebdriverIO: The main advantage is support for both web and mobile by using Appium. You can use WebdriverIO to automate:
- 🌐 modern web applications written in React, Vue, Angular, or other frontend frameworks
- 📱 hybrid or native mobile applications running in an emulator/simulator or on a real device
- 💻 native desktop applications (e.g. written with Electron.js)
WebdriverIO can be run on the WebDriver Protocol for true cross-browser testing as well as Chrome DevTools Protocol for Chromium based automation using Puppeteer.
TestCafé: Because TestCafe only launches browsers, it can automate browser on desktop or mobile. Unlike WebDriver, you don’t need special version of browser and driver to prepare to run tests.
There is an automated waiting mechanism, which automatically waits for changes to take place and retries with checkups. TestCafé knows when to wait and what to wait for before a test can continue. When tests are run, a large pointer shows which element is being interacted with.
There is also an automated authentication scripts, which save and activate user roles to easily switch between different user accounts.
Another cool feature is mobile web app testing, which is testing a web browser via IOS or Android devices, without any workaround to use Appium additionally. You can connect your iOS or Android device to TestCafe as a remote browser by using a QR code: Remote Browsers. There are also third-party browser plugins for testing on mobile devices: testcafe-browser-provider-idevice and testcafe-browser-provider-android.
Playwright: It auto-waits for all the relevant checks to pass and only then performs the requested action. If the required checks do not pass within the given timeout, action fails with the TimeoutError.
Playwright can intercept network activity to modify or mock network requests during test scenarios.
Playwright Test runs tests in parallel by default, using multiple worker processes. Each worker process creates a new environment to run tests. By default, Playwright Test reuses the worker as much as it can to make testing faster. Should any test fail, Playwright will discard the entire worker process along with the browsers used and will start a new one. That way, failing tests can’t affect healthy ones. You can control the maximum number of parallel worker processes via command line or in the configuration file.
Playwright Test will retry tests if they failed. Either the maximum number of retries should be passed when running the tests, or should be set in the configuration file.
Screenshot functionality: It offers both full page screenshot and element screenshot.
Browser Support
Selenium WebDriver supports all major browsers (Mozilla Firefox, Internet Explorer, Google Chrome, Safari or Opera).
Cypress currently, Cypress has support for Chrome-family browsers (including Electron and Chromium-based Microsoft Edge), and Firefox.
TestCafé supports multiple browser, including mobile versions. Google Chrome, Internet Explorer (11+), Microsoft Edge, Mozilla Firefox, Safari.
WebdriverIO supports also all known browsers such as Google ChromeDriver, Firefox — Geckodriver, Microsoft Edge — Edge Driver, Internet Explorer — InternetExplorerDriver, Safari — SafariDriver.
Playwright has full API coverage for all modern browsers, including Google Chrome and Microsoft Edge (with Chromium), Apple Safari (with WebKit) and Mozilla Firefox.
Language Support
Selenium: Java, JavaScript, Python, C#, PHP, Ruby, Perl & .Net
Cypress: Cypress tests are only written in JavaScript.
WebdriverIO: WebdriverIO tests are also only written in JavaScript / TypeScript.
TestCafé: TestCafe tests are also only written in JavaScript.
PlayWright: JavaScript & TypeScript, Python, Java, C#
Popularity | Community Support
This feature is very important. Because if the community is huge, there is always someone outside ready to help you whenever you are stuck.
Selenium Web Driver: Selenium has a vast community of active users, and you can easily find answers to problems on the internet.
Cypress: Cypress is quite fresh on the market as it appeared in 2017, and its popularity is steadily growing. In addition to social platforms, Gitter Chat is available, where you can ask your questions directly. https://gitter.im/cypress-io/cypress
WebdriverIO: They offer on the other hand Open Office Hours to support users. However, it is one of the best option who wants to use selenium together with JS language. There are also discussion forums are available.
TestCafé: There are 2 books available, explaining about TestCafe automation tool. These books contain exclusive hints and insights. There is also one community blog, which contains articles written by TestCafe community members.
Playwright: Since it is pretty new, it might take time to build a community. There is spesific Slack channel reserved for the issues. Otherwise it can be followed from Twitter for the new news.
Documentation
All JavaScript Automation Tools provide more or less similar documentation, which gives user the opportunity to start directly writing tests. Even though you don’t have any idea what Cypress, TestCafé, WebdriverIO or Playwright is, just going through the documentation you can handle any situation that you might face while building a new project framework. Almost all of them provides good example projects. From the documentation perspective I haven’t seen a big difference among them, however I find Cypress and Playwright’s documentation more user-friendly and more comprehensive.
When it comes to Selenium, it is hard to build an automation framework just by following the documentation, since it is complicated compared to other automation frameworks. However, as I mentioned earlier there’s a huge community, which you can find almost anything that you need, and there are many open-source ready boilerplate to start with.
Syntax
I will try to demonstrate very basic login testing with each of these tools to show the syntax. As we can see below, the syntax is very similar with the main actions. It differs only when the scenario get complex. We can also easily recognize that except Selenium and WebdriverIO, other tools provide auto waits, so that we don’t need to write wait actions explicitly, which makes them more stable.
Limitations
There are mostly workarounds, whenever there is a limitation on the selected automation tool. As all we know, every new feature comes with its limitations. The more you go deeper, the more you find limitations based on your needs and expectations. I will just try to mention the most common ones.
Selenium Web Driver: It is not very fast and requires much more knowledge to set it up correctly. It is also only used for web-based apps, and has limited support for Image Testing.
Cypress: Does not run in Safari (2nd most used desktop browser). Cypress has also limited support for working with iframes and requires writing a custom code to handle it.
Additionally, opening and manipulating multiple tabs and windows is not supported in Cypress. Since Cypress runs in the browser, it will never have multi-tabs support. Just like with multiple tabs , Cypress does not support controlling more than 1 open browser at a time.
WebdriverIO: Must run with WDIO to debug, and since it is Selenium based Selenium server must be started independently. WebdriverIO incorporates Puppeteer as second automation driver tool allowing for the extra features, which other automation tools already have.
TestCafé: Because TestCafe do not control browser, you can’t actually automate all users actions. For instance, TestCafé can’t open new tabs or open a new browser window in incognito mode. There can be also some issues running tests on 3rd party servers or inside iframes. It’s also hard to debug possible issues as you don’t know how actually a web page is parsed to inject automation scripts.
Playwright: Playwright does not support legacy Microsoft Edge or IE11 (deprecation notice). The new Microsoft Edge (on Chromium) is supported. Additionally, playwright uses desktop browsers to emulate mobile devices only.
Conclusion
Applications become more complex as they are built on layers and entire networks of sub-systems, including UI and API layers, external databases, networks, and even third-party integrations. So, there is always a need for thorough testing to make sure that application works as expected. This ranges from unit testing to end-to-end testing of applications.
As we discussed, there isn’t a perfect tool that everybody use, instead there are many options, among them we have to choose based on many factors, such as our needs, the knowledge and experience of the testing team, and product /client requirements. Therefore, the requirements should be identified first, then the evalution on benefits and costs can be analyzed to make the final decision. All of these tools have their own advantages and limitations, which means choosing between them is subjective to the scenario they will be used for. Diversity increases the competition, and this allows us to encounter better quality tools each day. 😎
Hope this article helps you finding a right tool or at least gives an insight, which satisfies all your needs.
☕️ Happy testing! ☕️
You can follow me on Medium for more articles, connect with me on LinkedIn
Sources:
- https://www.selenium.dev/documentation/en/webdriver/
- https://applitools.com/blog/new-features-selenium4/
- https://www.cypress.io/
- https://webdriver.io/
- https://testcafe.io/
- https://playwright.dev/
- https://codecept.io/testcafe/#cons
- https://www.npmtrends.com/cypress-vs-playwright-vs-testcafe-vs-webdriverio-vs-selenium-webdriver
