End to end testing with Cucumber, Playwright and Selenium
There are several kinds of useful tests for an application, but of course, end-to-end tests (e2e) are the most important.
If we take a look at Mike Cohn’s concept, known as test pyramid it’s clear that e2e tests are the highest level test form we can produce for our applications, considering software as a black box where our main concerns are related to behavior and functionalities.
The knowledge of how our system must work and the possibility to register and check it in an “automated way” can be useful when introducing new features/developments and we want to be sure to avoid regression problems. In addition to this, automated tests can be useful also if we want to simulate a load test.
One interesting approach can be to define application behavior, describing the interaction between the user and the browser. There are many frameworks and libraries we can use for these tests and BDD with Gherkin syntax, as seen here, can be a great solution to describe functionalities.
Gherkin and the example project
Let’s imagine our system has a frontend Angular application, which can invoke services from several backend endpoints (we don’t care about this). We will use an example login/registration project coming from this article and it’s available inside this Github repository with all sources related to this article.
We want to describe some scenarios related to the user registration functionality
Gherkin syntax required to describe this sort of scenarios is pretty simple, as you can find from the next feature file, where we created 4 scenarios when registration doesn’t work and 1 where everything goes well
Selenium & Cucumber test implementation
As we said, there are a lot of tools to create e2e tests and in this article, we will use Selenium and Cucumber. Selenium, an opensource project born in 2004, is often considered the de-facto standard framework for end-to-end web testing nowadays.
We can find a lot of projects and extensions related to Selenium and, despite its age, it’s a very useful and complete tool.
Our test project will use Selenium to drive the browser, following the scenarios describe using Gherkin and executed by Cucumber. Let’s add some Maven dependencies to our pom.xml
Previously defined Gherkin scenarios will be saved within a feature file, used by Cucumber to drive the execution of the tests.
Now we can set up the Selenium engine, using a simple utility class that creates a WebDriver class instance. If we want to choose a specific browser, we must download into our environment its driver. In the following list, you can find browsers used in this project and the related env properties you must setup (also from the command line when you launch the final test program)
- Chrome : ChromeDriver (webdriver.chrome.driver=path/to/the/driver)
- Firefox: GeckoDriver (webdriver.gecko.driver=path/to/the/driver)
Using test-browser env variable you can choose which browser to use, the default choice will be Chrome
Adding JUnit After annotation we can choose something to do at the end of the single test execution. If our scenario is failed we save a screenshot from the browser, using Selenium, so we can later investigate.
The final step of our implementation is to use Selenium API to describe the interaction with the front-end application. You can write it from the scratch or you can also use Selenium IDE, a browser plugin.
My favorite approach is to use Selenium IDE to create a draft version and then I can refactor it
As you can see, our test implementation tries to replicate what is the user interaction with the browser. For example, when I want to be sure about the registration, I will check for the text “Registration successful” on the page.
Playwright & Cucumber test implementation
Using the same feature file we can use another interesting tool: Playwright.
This is a Microsoft open-source project, providing a complete API to create end-to-end tests. There are multiple bindings of this project and here we will use the Java implementation.
Compared to Selenium it’s a newer project, with interesting features and it’s probably more “compatible” with modern web applications but we have to evaluate the maturity of this project using it on some real implementations.
Let’s add the dependency to our pom.xml
Test case implementation will follow the same “schema” we used for Selenium. So we must define a test class to match Cucumber with Playwright
there is also the definition of the Setup class, where we choose the browser and setup some parameters
Playwright doesn’t have as a prerequisite the browser driver installation because if it isn’t available Playwright will automatically download it during the first test execution.
In addition to this, we have a very interesting feature (not available in Selenium): video registration of our tests.
Finally, we have to implement the test cases using Playwright API (it seems a little bit readable compared to Selenium, but it is a personal point of view)
We can now manage the failed test execution using the Playwright feature to record the video. If the scenario is failed we can add this WebM video to the Cucumber report
Let’s start testing
We must start the Angular application. You can enter the angular-9-registration-login-example folder, to create a Docker image using the following command
docker build --no-cache -t bddfe .
and then you can start it
docker run -p 80:80 -t bddfe
If you point your browser to http://localhost you should see the application working. Now you can launch tests (remember to define webdriver.gecko.driver or webdriver.chrome.driver in your env or you must pass this additional parameter only for Selenium)
mvn install -Dtest-browser=firefox
You will see some “browser windows dancing” on your computer and, after the execution of the tests, you can control the results using Cucumber reports in the target/cucumber_reports/ folder
We used two different end-to-end test engines (Selenium and Playwright) with the same feature file so we will have two different Cucumber reports.
If we want to simulate an error we can change the check on the string “Registration successful” to “doh” e launch again the tests. In the Selenium report, we will see a screenshot
and in Playwright we will have a video of our problem
Conclusions
Test project implementation used in this article shows the possibility to use a library to describe end-to-end tests (Cucumber) and other tools to simulate these test using a real browser driver (Selenium and Playwright).
These libraries also provide different language implementations, so you can recreate this mashup also using Typescript, Python or C#.
Inside this repository you can find the source code related to this article.
