Why we favor Playwright over Selenium or Cypress
by Patrick Döring and Kenji Brueckner
ProSiebenSat.1 Tech Solutions (PTS) supports multiple web applications from the entire company. With that, there are several various requirements for the chosen UI testing tools. In our environment, we deal with simple single-page applications but also with complex distributed systems with one or more web interfaces. If we consider the whole value chain of different applications in combination to fulfill business requirements, it gets even more complex.
Some applications come with their own authentication integration; others rely on cloud services like Microsoft Azure Active Directory. In some user interfaces, we experience complex user interactions for example drag and drop operations or native browser alerts and dialogs. Additional challenges we face on web automation are slowness of test execution, unreliable wait scenarios and cross-origin redirects.
It is worth to be mentioned that some of our software engineering teams mainly use Java while others use JavaScript or Python stack. Our mission is to enable all teams to operate and automatically test the entire application within their selected tech stack.
Considering this, we want to talk about why we chose Playwright over Selenium and Cypress to tackle these points.
The considered tools
All three candidates — Playwright, Cypress and Selenium — are well-known and established open-source testing frameworks. In general, they are all comprehensive solutions and good choices to go with for automated testing — but in detail, they differ a lot.
- Selenium is a collection of different tools for UI test automation. It offers solutions to scale testing for all major browsers in different programming languages such as Java, JavaScript and Python. Among the evaluated tools, it is the solution that has been on the market for the longest time.
- Cypress, unlike many automation tools, is not built on top of Selenium and therefore is not limited by selenium capabilities. Its focus is on end-to-end tests. Tests are solely written in JavaScript. Cypress is a full test suite paired with its own runner and reporting. It supports main browsers like Firefox and all Chromium based browsers such as Chrome, Edge and Electron Apps. The tool comes with the ability to record videos and screenshots during test execution.
- Playwright, just like Cypress, is not built on Selenium but is a completely standalone solution. It supports all major browsers and different programming languages like Java, JavaScript and Python. Screenshots, videos and execution traces are supported out of the box. It comes with its own runner but can also be used with the runner of your choice. Playwright allows multi tabs, origins and users within one test.
What about our requirements?
Runner solutions
Cypress and Playwright come with their own runner solution, while Selenium needs to use an external runner. In addition, Playwright has the adaptability to operate with third-party runners such as JUnit or Jest. This is not available with Cypress.
Given the most flexibility and our desire to utilize runners which are already in use in other test levels, we consider Playwright to be the best fit.
Cross-Domain Support
First, we want to answer the question: Why do we need cross-domain support in our tests? The answer is simple: Because of OAuth2 authentication flows. Many of our applications authenticate against authentication providers such as Azure Active Directory. Cypress does not support cross-origin by design while Selenium and Playwright do. This makes the latter two to be a better fit for that requirement.
Multi Tab Support
Unlike Cypress — both, Selenium and Playwright support tests with multiple tabs. Hence, Selenium and Playwright are a better fit.
Performance
We ran some quite simple internal benchmark tests to gain an idea of which solution would perform best. In our tests, Playwright was the fastest, followed by Cypress and Selenium. It is important to mention that our tests have been basic and are therefore most likely not representative for every scenario.
We also had a look into external performance comparisons such as checkly. With the insights provided by the report, we considered Playwright to be the best choice for our needs among the tested solutions.
Drag & Drop Support
All three tools offer solutions for drag and drop. The most complex solution is Cypress. Here, you have to write a custom function that triggers a few mouse events in the correct order (click and hold, move, drop).
With Selenium it is a little less complicated: You have to programmatically define a drag item, a drop source and create a custom action that moves and drops the item to its destination.
The easiest approach is the one provided by Playwright. It offers a simple function with the parameters draggable selector and drop area selector. Since Playwrights approach is the easiest one to use, we consider playwright to be the best fit in that category.
Dynamic/fluent wait
Dynamic wait for us means the capability to wait until an event can be resolved as true. If the event cannot be evaluated true within a predefined timeout, it will be false.
Cypress and Playwright are designed around the fluent wait pattern. All their API commands make use of it when applying for working with elements. Selenium offers ways to achieve the same but with way more code produced. Therefore, Cypress and Playwright are the better choices in that sector.
Static wait
As static wait, we understand the capability to wait for a defined amount of time without any further processing of the test execution. Selenium and Cypress come with solutions for this out of the box. Even though it is possible on Playwright as well, you would need tricks to achieve that. Selenium and Cypress are better fits here.
Element Selectors
That category focuses on element selectors for UI elements. All three solutions support xpath, CSS, text and custom selectors for simple locating elements. Also, it is possible to combine selectors easily in all tools. Therefore, they all meet our needs.
Parallel test execution
To speed up test execution, we strive for a solution that can run tests in parallel. Playwright and the pro version of Cypress can do so. Selenium can achieve that by using third-party solutions such as TestNG. If we want to run tests in parallel for Cypress without a pro subscription, there is the open-source project Sorry Cypress. All tools can fulfill the requirements.
Video recording and screenshots
For better documentation and less flakiness in interpretations of test failures it is desirable to have screenshots and even test execution recordings. Cypress and Playwright are shipped with these capabilities out of the box. For Selenium there are extensions available. Therefore, all tools are a good fit in this category.
Running it in combination with AWS Lambda
Since we want to use the power of the cloud, it is a bonus if the chosen solution can run in the AWS cloud. All the tools mentioned are capable of doing so, which is why they all are an equally good fit.
Technology stack
While Cypress only works with JavaScript, both Selenium and Playwright have APIs for multiple languages. In our case the languages necessary are Java, JavaScript and Python. Given that Selenium and Playwright are the better choices here.
Overview
Conclusion
Even though Playwright is a relatively new player on the market — and probably the youngest among the evaluated tools – it meets our requirements best. It is able to overcome challenges where the other tools struggle to provide easy solutions. The support for all our programming languages makes it easy for us to exchange experience between different development teams, even if they are on different tech stacks.
In conclusion the choice was easy. Playwright is by far the best fit for our needs and therefore the tool of choice. We decided to use it in combination with external runners like Java with JUnit and JavaScript with Jest. This way we are able to use the runner engines as on other test levels (e.g. unit tests) with the same test report structure.
Nevertheless, it is fair to say that the other tools are still worth evaluating if you have other needs than our company. We used — and still use — them on a lower scale for some of our projects.
