Migrating from Nightwatch and Cypress to Playwright Test Framework

Nightwatch:

Our journey began as we transitioned from PHP to the Serverless Framework, crafting our in-house Donate platform. This marked the commencement of our utilization of the Nightwatch Test Framework in May 2018. Nightwatch.js, a free automation test framework written in Node.js, leverages the W3C WebDriver API, streamlining the creation of end-to-end tests in Javascript for our web-based apps and websites. It communicates over a restful HTTP API with a WebDriver server, making it one of the most popular test automation frameworks.

Departing from PHP behat tests, Nightwatch served us well by swiftly executing tests. The decision to use Nightwatch stemmed from the desire to align the programming language of our application with our entire tech stack (JavaScript and Node.js) at the backend, front end, and in automated tests. Its syntax was straightforward, utilizing Selenium for DOM element operations and providing clear, easy-to-use assertions and commands.

Integration with Browserstack enhanced our testing capabilities, allowing cross-browser and version compatibility checks. However, challenges arose as our application and testing suite expanded, especially when dealing with backend testing and third-party API dependencies. Nightwatch became less maintainable, prompting the search for a more efficient testing framework after Red Nose Day 2022. In case of a failure, reviewing the videos to identify issues was a straightforward process. The syntax was easily comprehensible. However, as our application expanded, so did the testing suite, especially concerning backend testing to ensure frontend data ingestion into the database. To accommodate this, I introduced numerous waits. Testing across multiple providers and reliance on third-party APIs led to prolonged test durations, occasional flakiness, and the need to either disable or fix tests via pull requests.

While the tests remained intermittently flaky yet manageable, challenges emerged upon implementing a Serverless Service for our ERP system. This service, which is responsible for executing actions from our Supporter Event Service to create and update records in ERPNext, required thorough checks to ensure accurate data storage across various applications. We implemented lengthy timeouts to counteract delays caused by extended data processing times.

Cypress:

Running Nightwatch tests on staging alongside Concourse pipeline-triggered end-to-end tests, we incorporated Cypress tests into pull requests with Github actions. Cypress offered advantages like automatic screenshots of failures and commendable documentation, aiding us during installation and initial test creation. However, its limitations included inflexible backend testing, limited iFrame support, and constraints on driving two browsers simultaneously. The lack of parallel testing support also posed challenges.

Playwright:

After the challenges faced, particularly after Red Nose Day 2022, I dedicated time to investigating a testing framework. The goal was to find a fast, reliable end-to-end testing framework that could effectively address the issues encountered with Nightwatch and Cypress.

Addressing the challenges encountered with both Nightwatch and Cypress, we discovered Playwright. After thorough research and experimentation, we found Playwright to be robust and faster than its counterparts. Implementing Playwright application by application proved seamless, with straightforward setup, configuration, and test examples. Integration with Browserstack for staging end-to-end tests allowed easy tracking and video viewing on test failures, offering flexibility to switch browsers. Playwright’s device emulation is another feather in its cap, allowing us to emulate various devices and screen sizes. This capability ensures thorough testing for responsive design, a critical aspect of our applications.

Example:

Advantages of Playwright:

One noteworthy advantage Playwright realized after transitioning from Nightwatch and Cypress is the significant reduction in test duration — from 1 hour to under 10 minutes for some applications. This efficiency has streamlined our code delivery into production, ensuring reliable tests even with frequent deployments. Our CI/CD pipelines consistently remain green, showcasing the reliability and effectiveness of Playwright in our testing workflows.

Opting for Playwright over Nightwatch has proven to be a strategic decision, significantly elevating our capabilities in web testing and browser automation. This robust automation framework brings forth a multitude of advantages that have reshaped our testing approach.

One of Playwright’s standout features is its automatic wait handling, as it intelligently waits for elements to be ready before interaction, reducing the need for explicit waits in our test scripts. This contributes significantly to the stability and reliability of our tests.

Playwright offers impeccable cross-browser support. Whether it’s Chrome, Firefox, or WebKit, Playwright ensures a seamless testing experience across various browsers. What sets Playwright apart is its utilization of a unified API, simplifying automation efforts by providing a single interface for multiple browsers. This not only streamlines our code but also enhances maintenance efficiency.

Playwright’s flexibility is evident in its support for both headless and headful test executions. This adaptability allows us to cater to different testing environments based on our needs. Moreover, the framework introduces a paradigm shift in test execution speed by enabling parallel execution of multiple tests. This not only expedites our testing processes but also contributes to a more efficient testing workflow.

The framework’s support for creating and managing multiple browser contexts adds a layer of isolation for different sets of tests. This not only enhances modularity in our testing approach but also fosters test independence.

Playwright’s commitment to comprehensive documentation and active community support further solidifies its position as a go-to automation solution. Access to resources, support, and updates on best practices and new features is readily available, making our journey with Playwright both seamless and enriching.

Integration with popular test runners like Jest and Mocha is the icing on the cake, ensuring Playwright seamlessly fits into our existing testing workflows.

In conclusion, by embracing Playwright, we have not only embraced a powerful automation framework but also revolutionized our testing landscape. Playwright’s array of features has not just streamlined our testing processes but has also empowered us to navigate the complexities of web testing with confidence and efficiency.