Why TestCafe?
In one of our recent projects at my company, we started using TestCafe for end-to-end UI automation. This article will describe how we ended up choosing TestCafe as the choice of framework for front end automation.
Let’s now look at the criteria for the selection of the framework and see how TestCafe provides an answer to these.
Development Language
There is no challenge to the fact that Javascript is the language of the web and which means that using the same language for test automation will have lots of advantages.
In our team’s case, we decided to go with TypeScript as the development language since we all agreed that that is the best way to write clean and maintainable code. This means that both the Dev’s and QA’s can now contribute to build and maintain the automation codebase. Yes, that is a great way to push for ‘Quality as a team’ philosophy.
Using Typescript will help the team re-use many of the API calls defined in the UI codebase for automation when certain pre-conditions need to be set for the test using API calls. If the language is different, it would lead to unwanted duplication when creating utility functions and API calls in the automation code. An example of this could be to write API calls to generate a user token, delete entities as part of the test teardown, etc. In most cases, all these API endpoints will already be called by the front end application which could be reused for automation.
Another beauty with the choice of language is the ease with which one can integrate into the development workflow.
You do:
yarn build — to build the application
yarn test — to run unit tests
yarn start — to start the app
yarn e2e — to run the end to end UI tests
Easy Setup
npm install testcafe — And the world of automation is all yours
Very easy to set up and can start writing actual code within minutes. No installation of web-drivers and no additional libraries. TestCafe pics up the browser installed on your machine and runs the tests on them.
Want to run tests in parallel? No worries TestCafe supports them natively!
Browser Support
Let’s be open about this. All of us hate IE!
But unfortunately, there are still a lot of companies using IE and that means we still need to test our software in IE and our frameworks should support the same.
The case is the same with our company and our data showed that we have the most users using Chrome followed by Internet Explorer 11, Firefox, and Edge.
Thankfully TestCafe supports all of them including Safari and mobile browsers (Chrome & Safari mobile). It also supports chrome device emulation.
On a side note, this is also one of the main reasons we had to look away from Cypress, which is also a cool kid in the block for test automation. But they do not support IE!
How cool is it if you could scan a QR code and the test runs on your phone? It is that-easy to run tests on a mobile device with TestCafe. And of course, there is support for standard remote execution as well.
The Legendary waits (Flakiness)
This is an awesome feature to have if you come from a Selenium background where you have to carefully place the right waiting mechanisms to make sure that the automation does not break. Even though one can argue that if you have the right framework on top of Selenium and scripting practices, you could still manage to write less flaky tests, it is still a treat to have everything handled by TestCafe out-of-the-box for you.
Another cool feature is the “Quarantine Mode”. This mode re-runs failed tests to confirm that the outcome is consistent and deterministic. If the results of the re-runs are mixed then the tests are flagged as unstable. How cool is that!
HTTP Mocking
Have you ever automated or even tested a scenario where your API returns a 500 and the UI needs to handle it accordingly?
It is tough to do this since you may need to manipulate your backend service to fail to test the behavior from the UI. Or you need a proxy setup or a framework that can read the backend calls of your web application and change the API response.
TestCafe has this HTTP mock functionality inbuild and ready to use.
One can easily intercept API calls and respond with the desired status, headers, or data. This means one can easily test edge-cases or simulate API failure scenarios. Another advantage of using mocks is to speed up the development time since the mocks are going to respond faster than your actual backend service. You could also run your tests against a mock service if the dependent service is not completely under your control and may cause non-deterministic test results.
Let’s Debug the Bug
UI tests are not easy to create. One has to run the tests multiple times, make changes, re-run and the loop continues until we have the final product. This means that the framework has to support good debugging methods.
There is a quite informative console error when something goes wrong during the execution. Along with the reason for failure TestCafe prints out few lines of code where the test failed so that the user can instantly get an idea of what could have gone wrong.
TestCafe also catches any javascript error which might come up on the website so that nothing skips the tester’s eye.
TestCafe provides screenshot and video recordings for test failures out-of-the-box.
If these debugging methods are not good enough one could use the Visual Studio Code debugger or even debug in Chrome developer tools.
One thing which is missing from TestCafe is the Time Travel feature in Cypress which is a really cool way to debug and understand the test steps.
CI Pipeline Integration
Like most new-age test automation frameworks, TestCafe too has easy integration with CI providers. Since TestCafe does not need any ‘drivers’ like selenium to run tests, the only pre-requisite is that the required browsers should be installed on your CI server and you are ready to go.
We use GitHub Actions in our company and the ‘windows-latest’ VM provided by GitHub already has the latest version of Chrome, Firefox, IE & Edge always available, which makes it very easy to run tests on CI.
Cross Browser Grid Support
When the regression suite grows there comes a point when we would need the support of 3rd party cross-browser testing platforms to scale our tests. TestCafe users have that covered as well. Popular providers like SauceLabs, Browserstack, etc have TestCafe integrations to run the tests on their services.
Support for Visual UI Testing
Even though TestCafe does not have any inbuilt support for visual test automation, major 3rd players like AppliTools and Percy support visual testing with TestCafe
What is not that great?
TestCafe does not seem to have the functionality of logging individual test steps into its report (at least I could not figure out a way to do this). This means when a test is passing, you don’t see any additional details of the individual steps on the tests.
However, when a test fails we have sufficient information to figure out what went wrong along with the screenshot of failure.
TestCafe does not have direct support to use XPath to define selectors. What you can do have is a more descriptive selector like:
Selector(‘some css selector’).child(‘some css selector’).withText(‘some text’)
Even though I like it, it might take some time to get used to this.
Let’s Conclude
For us, it’s early days with TestCafe and it looks great till now. With all the various in-built features, it very easy to concentrate on automating tests rather than adding framework features to automation.
What we would like to wait and see is how stable TestCafe is when we run larger test suites and how well it auto-handles waits and thus reduce flakiness.
If you are looking for alternatives to Selenium and happy to start learning TypeScript/Javascript, give TestCafe a try.
Till the next blog post! Adios!
