Optimising test efficiency: a Playwright approach
Frameworks come and go but we always have the discussion about testing. This leads to the question, why is testing important?
Test automation reduces the time and effort that go into executing manual validations required for each deployment exposed to clients. Code generation of Playwright allows teams to record test scenarios instead of manually implementing them.
The answer that come to mind is trust. Trust for me, as developer, to make changes in a codebase that will keep the system up and running and even more importantly is the cause of this action, the trust of clients.
The concept of product mindset teaches us that individual blocks, for example an API and a frontend, have value for the customer only when interacting in form of a product serving a need. We want to test end to end the product, similar how a client is using our product, to get the real value.
Let’s start with a plan, we will call it test catalogue. There are 3 main categories to test:
- happy scenarios | the user is able successfully finish the flow;
- cancel scenarios | when the user on purpose doesn’t what to continue;
- error scenarios | in case of technical or functional errors.
Once our product is live and there is a need to add a new functionality, a bug introduced by a new feature, affecting the existing features, it’s called regression bug. Here’s a fun fact on the topic: Software issues are called ‘bugs’ because back in the time when computers were the side of a building, and could not fit in our pockets, one little real bug managed to burn a circuit board. The engineers called the defect ever since, a bug.
Cost and importance of testing in lower dev environments
If a defect is find at the moment of drafting the product or in a lower development environment, the cost of fixing is much lower.
For each deployment exposed to the clients to ensure resilience all test scenarios must be validated. All CI\CD tools enable us to have a rapid time to market, starting from the moment when the code is written, until it is used by a real client as a feature. Manually testing will require many hours, but now with the raise of AI is clear that time spent for creativity and critical thinking is more valuable to humans. And let the automation of testing unblock this time.
Introducing Playwright
For web testing and automation we will use Playwright, a framework designed to allow us to run end-to-end tests reliable in all modern rendering engines (Chromium, WebKit and Firefox). It offer the possibility to emulate native mobile like Mobile Safari and Chrome for Android.
Playwright API can be used with JavaScript, TypeScript, Python, .NET or Java. In the examples bellow, JavaScript will be used to install run in a terminal:
>> npm install playwright
One of the situations which was problematic in the past when you wanted to write an end-to-end test was the wait for elements to be present in the DOM. Having a scenario like:
‘As a user when I open the browser and navigate to https://www.w3.org/
Then I click on the button “Read more about W3C”
I expect to be redirected to about page.
Depending on the performance of the web page and the network conditions(eg. you are in a train and suddenly network connection is gone for few seconds), the button will not be yet present in the DOM; one way in the past was to set a timeout for the testing tool until the element was present.
Playwright is leveraging auto await feature of JavaScript and the code to select the button is clean and avoid flaky tests:
A flaky tests is defined as a tests that is passing or failing without changing anything in the code.
Having a suite of test scenarios, Playwright ensure each test runs in isolation by creating a new browser context, equivalent to a new browser profile.
Record tests using code generation
The code for the test above can be generated by Playwright. Let’s do it:
An incognito browser is opened, and Playwright Inspector is generating the selectors and actions based on the action we perform. By hovering on the ‘Read more about W3C’ element we can see the selector Playwright will add in the code.
All the actions performed by the user will be recorded as code.
Modify API response for error scenarios
In our test catalog, we defined error scenarios that must be validated. Most error scenarios require that accounts or the data must be in a specific state. This is not always straight forward to accomplish.
In the suite of mock helpers, Playwright offer the possibility to modify API responses.
Manual vs automated ?
Imagine you have 24 deployments in a year where 100 test scenarios must be validated. If we compare the effort for manual testing and automated:
The estimation to implement test automation for a scenario is 2 minutes. For 100 tests we have 200 minutes, running in CI the tests don’t need more time from a dev for the rest of releases.
If we make the calculation, in a year there are 77 hours saved for an engineer, which can be approximate to 2 weeks of work.
Testing is influencing the time to market. If we have automated tests the time spend is <1min.
Having code that is running the tests is enforcing the confidence checking of our execution. Manual tests can introduce human errors.
Takeaways
1. Test automation reduces the time to market by catching regression bugs early in the development process. Compared to manual testing where engineer input is needed every time a test is executed, using test automation the test is written once and executed multiple times even automatically in CI which lead to saving of engineers time and enable reusability.
2. Planning the test process using a test catalogue sets the ground for test coverage.
3. From our estimation, test automation can save 2 weeks of an engineer’s time / year on a single asset.
4. Codegen feature of Playwright is a boost for developer productivity when the test catalogue must be implemented. Performance wise Playwright is ~4x faster than Cypress.
Playwright has excellent documentation available at https://playwright.dev/
