Playwright: everything else is wrong
This is a summary of my research to find the ultimate tool for end-to-end testing for modern web apps and modern devs team.
TL;DR
Playwright is the new star for test automation, and it’s due to its very likable features and its frequent releases shipping awesome new testing opportunities. Its best features are:
- modern browser: Chrome, and Safari and more
- language: Typescript, Python and more, well known by devs
- ease of use, readability and useful assertions
- powerful features on the report that makes debugging super easy
- Strong integration on your CI/CD
- A component testing in alpha that looks promising [1]
I was looking to start a new test automation project for my new company exaprint.fr. I had the time to look for new tools on the market, and didn’t want to rush to reuse Selenium x Gauge that I liked and known well (see previous posts [2]).
I wanted the right tool for our team, knowing I would need to onboard the development team with me. And so began my research to find the best test automation tool.
My research lead me to choose between two contenders, with the main goal to have a testing tool that can be used, and understand by the devs.
On one side, we have Playwright, an end-to-end web browsers automation tool (since 2020), open-source and maintained by Microsoft, by the Puppeteer team.
On the other side, we have Cypress, an e2e web browsers automation tool (since 2015), open-source and maintained by Cypress.
Comparison points
Community
Advantage to Cypress, which has a larger community than Playwright. However Playwright, based on puppeteer, can use it as a basis for help.
Playwright, is now before Cypress on the not necessarily relevant criterion of stars on Github and gain adoption.
Supported languages
Draw, even if there is a Playwright advantage on the number: TS, JS, Python, .NET and Java. While Cypress is just TS, JS.
As it is a criterion where we just need the right supported language (TS for me), draw.
Supported browsers
Advantage to Playwright, which supports Chrome, Firefox, Safari, Edge and Opera. Cypress is Chrome, Edge, and Firefox. Safari is still a WIP [11].
Syntax and readability of tests
?, hard to assess. From the opinions available, Cypress is easy to understand for QA or junior devs, while writing tests on Playwright, with its more JS approach, may speak more to the more experienced dev [3], [4], [5].
From my experience with Playwright, the documentation is excellent and very useful, and the readability can be very good, if you try to write good code!
Debugging
Advantage to Playwright. It is possible with Playwright to run step-by-step a test, with the browser open alongside.
It is also possible to have a complete test execution report including screenshots and/or video of the execution if it has been configured as well as the possibility of returning to each step of the DOM during the test thanks to the Trace Viewer, we can see the before/after of each action, observe the DOM at that moment, see the console and the network activity at each step. You can’t imagine how useful it is if you have not used it!! :)
On the Cypress side, it is possible to launch the tool and see the test running live with the test information (cypress open), and if the test fail, the interface remains open for debugging. A nice feature: a hot modification of the test automatically restarts it.
Cypress also allows to have a behavior similar to Playwright and its Trace Viewer when the test is played live.
But, the advantage for me is lost here by Cypress because this very practical possibility is not available in the report. So if a test fail during a build, we would not have access to a lot of information, and we would have to restart the test, hoping it to fail again.
Parallelism
Advantage to Playwright. The ability to run tests in parallel is native to Playwright. You can choose the number of “workers” you want in the config, to adapt to the CI, and it is faaast!
With Cypress, the feature is not recommended (!) locally [4]. In CI, you must have an account on the Cypress Dashboard (can be a security aspect to send the reports there), and the free account limits to 500 recordings per month [6], it is not much!!
Execution time
Advantage to Playwright. On the tests I run to compare, Playwright was quicker. It should nevertheless be noted that on other tests, the results vary easily [3], [7]. And the advantage is with parallelism, unbeatable!
Limitation of Cypress [8]:
By design, no cross-origin test management (we need to stay on the same domain during a test).
No multi window, no multi tabs.
No possibility without a complicated hack to work on pages with iframe or to upload documents [9], [10].
The other tools I looked into
Selenium
Probably see by many as the reference for automating web browsers, Selenium is here since 2004, has a huge community and is quite stable and well documented.
“Selenium automates browsers. That’s it!”
That being said, you understand that it is possible to use Selenium for end-to-end testing on web based application, because you will be able to automate user action on the browser.
The problem to adopt Selenium is that it is quite hard to use at first, and it has a steep learning curve, that make it a tools that mostly QA engineer use, and if you want your e2e tests to be done by the developers, or at least have them participate, Selenium is not the best tool.
Robot Framework
Another popular tools in the QA community. It’s here since 2008 and is widely used. The problem is, it is not a tool to automate web browsers, and must be used on top of another tool like Selenium.
Robot Framework, give the user the possibility to write tests in a clear syntax, with human readable keywords.
The problem to adopt Robot Framework is that you still have to use a tool like Selenium to automate browsers and have e2e tests.
Gauge
A less popular tool for test automation, that is here since 2015. Like Robot Framework, it does not automate browser, and need to be use with another tool like Selenium.
It give the possibility to write very clean and readable tests. I used this tool before and recommended it, but having to use Selenium was the reason to not choose it.
Cucumber
A tool for BDD (Behavior Driven Development) testing with the Gherkin language (Given/When/Then).
But then again, it is not a browser automation tool, and the problems with Selenium remains.
Puppeteer
A Node.js tool that controls Chrome with the DevTools protocol (2017). it is not an e2e testing tool but just a browser automation tool (eg there is no assertions without a plugin).
It only works with Chrome, in JS. Works with promises to wait for each action. Interesting for the devs [3].
The non-testing tool aspect, and the missing cross-browser testing aspect, is a problem to adopt Puppeteer.
[1]: https://playwright.dev/docs/release-notes#version-122
[2]: https://medium.com/@jeremie.fleurant/loutil-gauge-pour-l-automatisation-de-tests-1aa11c3d95f4
[3]: https://dev.to/jalal246/to-puppeteer-or-to-cypress-4ccl
[4]: https://alisterbscott.com/2021/10/27/five-reasons-why-playwright-is-better-than-cypress/
[5]: https://medium.com/sparebank1-digital/playwright-vs-cypress-1e127d9157bd
[6]: https://github.com/cypress-io/cypress/issues/2520#issuecomment-485356759
[7]: https://medium.com/comsystoreply/cypress-vs-testcaf%C3%A9-3b26cfb803f3
[8]: https://docs.cypress.io/guides/references/trade-offs#Permanent-trade-offs-1
[9]: https://www.cypress.io/blog/2020/02/12/working-with-iframes-in-cypress/
[10]: https://github.com/abramenal/cypress-file-upload
[11]: https://github.com/cypress-io/cypress/issues/6422
