As a Software Development Engineer in Test, I actually enjoy reviewing test automation tools.
We use tools because we want to get the job done, without losing our minds or breaking the bank.
But how can we make sure that we’re evaluating a set of tools correctly?
We need a clear set of objectives, and to measure how each tool can help us reach those objectives.
The 3 objectives:
✔️ Ability to create automated tests for a complex web application.
✔️ Ability to execute these tests on all major browsers.
✔️ The tests should be easy to maintain.
Most of us have these objectives when evaluating a test automation tool.
1. Endtest
I really enjoy working with Endtest.
It’s part of a new generation of Intelligent Test Automation tools.
That means that you don’t have to stitch together different libraries and services in order to create a test automation framework.
Worth mentioning that it has a built-in test recorder.
After you create the tests, you can run them on their cross-browser cloud:
A pleasant surprise was that I’m able to even execute the tests in different Geolocations around the world:
A test case will be made up of different steps.
Each step is defined by an action, and there’s a long list to choose from:
It seems to have all the building blocks of a scripting language:
- variables
- reusable components
- if statements
- else statements
- loops
- calls to external services
- database connections
Having all those options means that you should be able to automate any scenario.
On my side, I was able to automate even a Sign Up flow that requires email and SMS validation.
In our web application, we have a number of iframes and we even need to switch between multiple browser tabs.
Endtest was able to handle those scenarios.
Here’s what the results look like for a basic test execution:
And yes, it ticks all the boxes:
✅ Ability to create automated tests for a complex web application.
✅ Ability to execute these tests on all major browsers.
✅ The tests should be easy to maintain.
I’d say the strongest point is that the tests are easy to maintain, but I also like the fact that it’s fast.
2. Selenium
Selenium was the first test automation tool I ever heard of.
And this was back in 2012.
I’ve used it for a number of years and had good results.
Selenium works with the WebDrivers developed by the browser vendors themselves.
That means you have to ocasionally download and update your chromedriver, geckodriver, edgedriver, etc.
I love the fact that Selenium tests can be executed on all major browsers, including Safari.
Selenium can be coupled with Selenium Grid.
You get to decide if you install all those browsers locally or if you use a Selenium Grid provider.
One disadvantage of using Selenium Grid is that it will make your tests slower, since each request will consume time to go through the network and to be processed.
Selenium can perform any action related to web elements and pages, but it can’t send API requests or connect to databases.
It also lacks some of the built-in components that Endtest has, like the functionalities for email testing and SMS testing.
That means you’ll still need to stitch together some libraries and services in order to perform end-to-end testing.
The major downside of using Selenium is that you need to write a lot of code, and that can consume a lot of time and generate a lot of discussions.
Due to that fact, it seems Selenium doesn’t fully check the last box:
✅ Ability to create automated tests for a complex web application.
✅ Ability to execute these tests on all major browsers.
❌ The tests should be easy to maintain.
3. Playwright
I love the fact that Microsoft started investing more in open source technologies.
And Playwright is a prime example.
Playwright uses browser remote debugging protocols (e.g. CDP for Chromium) over the process pipe.
The syntax is a bit easier to use than Selenium.
By using the Playwright API, you can interact with the browser and elements from a web application.
It also support mocking, which is really cool.
You can intercept and parse requests with it, but you can’t send requests.
That means that you can’t really use it to properly test an API.
I do think the Playwright Inspector is really cool:
You’ll still have to paste together different libraries in order to get an end-to-end testing flow.
If I was doing this review 1 year ago, I would have been upset that Playwright doesn’t support running tests on Internet Explorer 11.
But since Internet Explorer 11 is no longer supported by Microsoft, I can just forget about it.
Safari is somehow supported, in the sense that they support WebKit, which isn’t exactly the same as running a test on an Safari browser.
Here’s how it did:
✅ Ability to create automated tests for a complex web application.
✅ Ability to execute these tests on all major browsers.
❌ The tests should be easy to maintain.
Unfortunately, even with a simplified syntax, the tests are still a bit difficult to maintain.
4. Puppeteer
Puppeteer is a Node library that provides a high-level API to control headless Chrome over the DevTools Protocol.
Now, we know Google built some great products, like Google Search and Gmail.
But they also built tons of products that ended up in the Google Cemetery.
It’s still too early to tell in which category Puppeteer will be.
We can clearly see the mocha-flavored syntax.
Just like with most frameworks, we can identify the elements by using different attributes.
Our team played with both Playwright and Puppeteer, and I wouldn’t see a clear reason why someone would choose Puppeteer right now.
The lack of community support for Puppeteer might be an issue, since you may struggle to find answers on StackOverflow or Reddit.
The cross-browser support is decent, but not great, since it only works on Chrome, Edge and Firefox.
Here’s how it did:
✅ Ability to create automated tests for a complex web application.
❌ Ability to execute these tests on all major browsers.
❌ The tests should be easy to maintain.
5. Cypress
I heard about the Cypress hype.
Is it really the Selenium killer?
The first blow was when I found out that I have to pay in order to use the full capabilities and put my tests in a CI/CD pipeline.
I might be old-fashioned, but when I see an Open Source library, my expectation is that I can use it for free, without any restrictions.
Turns out you can’t really use without the Cypress Dashboard, which is a paid component.
Unfornately, I wasn’t able to automate my test scnearios with Cypress, due to the following limitations:
- Cypress cannot test multiple domains in a single test
- Cypress cannot test in multiple browser tabs
- Cypress can’t reliably detect and interact with elements from iframes
This raised some questuons on my side:
I never saw anyone using Cypress for a complex end-to-end test.
I only saw folks using it for clicking on a few buttons.
So, here’s how Cypress did:
❌ Ability to create automated tests for a complex web application.
❌ Ability to execute these tests on all major browsers.
❌ The tests should be easy to maintain.
Don’t get me wrong, it might be great for Developers that only want to test a basic scenario on Chrome.
But I think it’s just not ready for real-life complex testing scenarios.
Other tools
If you know other tools worth mentioning, feel free to write about them in the comments.
I will do my best to reply to each comment.
It doesn’t matter if the tool you want to suggest is open source or proprietary, it just needs to get the job done.
