Benchmark for Low-code/No-code Test Automation Tools

Published in

adessoTurkey

10 min readDec 14, 2022

Introduction

Test automation plays an essential role in delivering high-quality software quickly. To ensure that this process can be integrated into agile, various tools came out to fill the gap. There are many products on the market with different strengths and also weaknesses. Since the use case determines if a product is good or bad, to create a fair ground for these different tools, test automation will be taken into account in general terms, and some of these tools will be benchmarked in this story.

The current test automation trends are also taken into account, e.g., the use of artificial intelligence and machine learning (AI/ML), which improves the capabilities for test optimization, intelligent test creation, execution, and reporting. There are also some challenges that come with test automation tools, these will also be addressed, such as:

Required technical knowledge
Required coding knowledge
Support system
Analytics
API and services testing
Cost

At this point it would also make sense to define what “low-code” means:

If a tool handles the runs of tests with an architecture that is in the background and presents a simpler interface that the users can use via pre-defined scripts, it is categorized as “low-code”.

Selenium IDE

Arguably, Selenium is currently the most popular test automation tool out there. It is an open-source test automation tool, which provides a lot of space for developers and testers if they have a little experience and skills in programming and scripting.

Test scripts can be created in a variety of languages (Java, Groovy, Python, C#, PHP, Ruby, and Perl) that operate on several OSs (Windows, Mac, Linux) and browsers (Chrome, Firefox, IE).

A big downside is that it doesn’t have any support system. Users have to find their own solutions through trial and error most of the time.

It does not have an in-built reporting system; an external reporting system needs to be used. It also doesn’t have any specific framework; an external framework is needed.

Finally, there is no internal object repository, so when the number of scripts is high, it might be difficult to manage all of them.

Tricentis Tosca

Because of its features, Tosca is usually compared to Selenium. Tosca is more of an enterprise tool because it is suitable for use in large-scale applications.

Tosca covers almost all testing activities, ranging from test design and test automation to test reports and analytics internally. It provides a simple user interface and a variety of features for planning, creating, carrying out, maintaining, and optimizing API testing.

Moreover, there is 24/7 support from Tricentis team, which saves a lot of time and work, just to look for a solution to a problem.

Some of its useful features are that API tests can be performed across browsers, mobile devices, and platforms. Multiple protocols and standards can be used (HTTP(s), JMS, AMQP, Rabbit MQ, TIBCO EMS, SOAP, REST, and IBM MQ).

UI test cases can be automated with zero coding/scripting, hence there is an object library that is ready to use, which quickens the process of testing. This, for example, shortens the amount of time dedicated to regression testing. Also, the tool works based on the linear expansion or LinearQ (SM) methodology, which also saves a lot of time during the test design process. Tosca platform definitely accelerates testing to be in line with DevOps and Agile.

In September 2020, the first version of Project NEO, Tricentis’ AI-powered test automation engine, was introduced, which will probably be the bread and butter of the industry soon.

Katalon Studio

Katalon Studio is a well-rounded automation tool for testing API, web, mobile, and desktop applications and also supports multiple platforms.

Katalon Studio makes use of the Selenium and Appium engines and offers an integrated environment for testers who struggle to install and integrate several frameworks and libraries in order to utilize Selenium and Appium.

A unique feature is that users can use in-built libraries or create custom models that fit their special case. No-code, low-code or full-code options are available in the tool, users can record-and-playback or drag-and-drop to create automated tests or use the script mode to build custom keywords and actions.

It has a unified IDE, where users have access to a complete library of web, mobile, API (supports both SOAP and RESTful) desktop keywords, which eliminates the need to jumping from tool-to-tool for different app types.

Katalon can be used both for automated and exploratory testing.

Using Katalon TestOps, users can schedule executions, view test reports and analytics, as well as manage test activities.

It has free and premium plans available; the free plan can be used to see if the tool fits the users’ needs.

With the update in April 2022, data-driven testing is now supported at the test case level and in dynamic test suites.

Appium

Appium is an open-source automation testing tool for mobile web, hybrid applications on iOS/Android mobile, and Windows desktop platforms. The tool supports multiple programming languages like Java, Ruby, Python, etc. and allows cross-platform (Android & iOS) testing on real devices and emulators.

Appium is mainly used for mobile automation and was designed to meet mobile automation needs according to a philosophy outlined in these points:

Users shouldn’t have to recompile the app or modify it in any way in order to automate it.
Users shouldn’t be locked into a specific language or framework to write and run tests.
A mobile automation framework shouldn’t reinvent the wheel when it comes to automation APIs.

From the last point, it is clear that Appium doesn’t come with fancy new tools; it gets the job done as an open-source automation tool.

The tool requires coding knowledge, so it would be more powerful in the hands of an experienced tester.

Testim

Testim is a test automation tool that utilizes the power of AI to automate the authoring, execution, and maintenance processes of web app testing, which also targets functional, end-to-end, UI testing and cross-browser testing.

Testim offers record-and-playback and a reporting tool, but test executions/runs are limited in the free version. Some of the key features are:

While beginners can start with the no-code mode, testers with programming expertise and developers can utilize the Testim Dev Kit.
Self-improving AI-powered stabilizers, eliminate flaky tests that devour resources.
Testim Visual Editor works in synergy with Smart Locators to record the manual steps and identify all the elements in your web application.
Tests can be grouped and reused to shorten test creation time, as well as test condition and validation.
Easily scaleable, Testim efficiently expands testing operations with control, management, and insights.
Data-driven testing can be done with CVS, JSON, or Excel files.

Cypress

Cypress allows complete end-to-end testing, with the tool, users can set up, write, run and record tests.

This low-code tool is open-source and “developer-friendly”, as it states. There are some features that sets Cypress apart from other tools, like:

Cypress takes snapshots as the tests run, the user can see what happened at each step with the help of Command Log.
Users can debug directly from familiar tools like Chrome DevTools.
The tool automatically reloads whenever the user makes changes to the tests and also users can see commands execute in real time in the app.
Cypress has automatic waiting feature, no need to add waits or sleeps to the tests. Cypress automatically waits for commands and assertions before moving on, which eliminates asynchronisation.
Users can easily control, stub and test edge cases without involving their server. The network traffic can be stubbed however the user likes.
The behavior of functions, server responses, or timers can be verified and controlled. The same functionality from unit testing is available to users.

Maestro

Maestro is a simple and effective mobile UI testing framework. It is built on learnings from its predecessors (Appium, Espresso, UIAutomator, XCTest) and allows users to easily define and test their Flows.

Flows are parts of the user journey in the app. Login, Checkout and Add to Cart are three examples of possible Flows that can be defined and tested using Maestro.

Maestro aims to overcome the flakiness. E.g., UI elements will not always be at their expected places, or a screen tap won’t always go through; Maestro embraces the instability of mobile applications and devices and tries to counter it.
Tests are interpreted, without the need to compile anything. The tool is able to continuously monitor the test files and rerun them as they change.
Maestro can be categorized as low-code since the syntax is declarative yet powerful.

Ddosify

Ddosify is a no-code load test automation tool, that allows its users to run regular load tests. Without load tests, businesses can’t grasp the importance of the load capacity of their systems. With Ddosify users can:

find out the maximum service capacity of the tested web system.
· start the test either instantly or on a future date.
define periodic load tests.
determine the origins of requests and the test durations with an easy-to-use web interface.
review past load tests and general performance from the reports and charts section.

Ddosify is multi-location-based, so the origin of the requests from 100+ locations can be adjusted.

With the use of detailed reporting, users can view insightful reports and charts to improve their service capabilities.

It is easy to use; the tests can be customized with scenarios on UI without a line of code.

Some of the upcoming features are:

API integration
CI/CD integration
Test scheduling

Ranorex

A selection of capabilities for web, mobile, desktop, and API testing are offered by Ranorex. Ranorex has the features of UI element recognition, editing, and management; it excels in desktop-based test automation.

It also has an intuitive GUI, record/playback, script generation and can also be integrated with Selenium Grid.

It has both paid and free membership options. Some of its key features are:

Record-and-replay, robust capabilities to build tests without programming
Full development environment, script in C# or VB.NET.
The tests can be run on a Windows desktop, all popular web browsers, and Android or iOS devices, including real devices and simulators/emulators
Tools like Ranorex Spy for machine-trained object detection, Ranorex Coach for test healing, video reporting of test execution and PDF validation
Test data can be pulled from CSV or Excel files, or a SQL database
Ranorex Studio tests can be added to CI/CD processes by integrating with Jenkins, Travis CI, and Git. It also has built-in integration with Jira and TestRail.

Leapwork

Leapwork is also a paid no-code test automation tool with some state-of-the-art features like “visual coding”. It has an intuitive, visual way to automate tests and processes, and intelligent building blocks that handle logic and give a graphic overview of every step. Some other features are:

Users can record their steps with the smart recorder feature, which allows users to create flows with their data, validate their tests, and just tell the tool what to automate, which improves productivity.
Leapwork allows users to reuse components; common steps can be changed into reusable sub-flows. Meaning less maintenance and fewer development bottlenecks.
Hypervisual debugging, users can quickly find the what and the why in three ways: video recording, data-level insights, and activity logs.
Leapwork can also run tests in parallel and schedule test runs.
It uses RestAPI, which makes it compatible with the most common DevOps tools and orchestrators users already use in their CI/CD pipeline.
Also supporting workflow and team management, users can see what’s ready to go, and what’s in review and draft mode, and track changes and roll them back with version control.
It is integrated with Power BI and Tableau, so users can make sense of data.

Conclusion

When selecting a testing tool, it’s important to consider future trends and advancements in addition to existing requirements. Basic optimization, test case automation, data creation automation, smarter solutions, and analytics should all be supported by a good tool.

For a simple piece of software, a complex system might not be the right choice, but for a large-scale project, a complex tool is needed to manage all the inputs/outputs of the project. The selected tool should allow testing on a large-scale project to be in line with DevOps and agile, and also bringing in state-of-the-art tools for its users.