AI-based Test Automation Tools
Part 3: AI Automation in the Wild
Part 1 of this article, “The Current State of AI in Testing,” can be found here
Part 2 of this article, “An Overview of AI-Based Test Tools,” can be found here
Part 4 of this article “AI and the QA Profession” can be found here
Quick Start
For the products I test, platform end-to-end flows are difficult to execute (even manually) because they require complex and thoughtful data setup, a deep understanding of the platform, and familiarity with the technical details. Add to this limited QA resources and challenging deadlines, and you can begin to see the problem I needed to solve.
You may recall that two primary problems motivating this research were (1) automation of complex platform-level E2E tests in the shortest possible time and (2) decreased time spent on maintenance. I read a lot of documentation, white papers, and watched video lessons for many AI-based automation tools. You can find the comparison table of the most popular and most promising AI-based tools in Appendix 1. I selected three of them (Testim, Mabl, and TestCraft) and automated one complex test scenario using each of these tools.
The software platform for which I needed end-to-end automation consists of three UIs, over 50 APIs, data processing pipelines (queues, databases), and has multiple dependencies on external data. Our teams are located in five geographical locations and four time zones. We’ve mastered testing individual components using unit tests, traditional testing tools (Cypress and a home-grown tool built from Cucumber and Java Script), mocks, and simulators for external dependencies. However, platform-level end-to-end tests were our pain point and I needed a way to automate them as fast as possible.
Because time was my most valuable resource, I concentrated mostly on time-saving features. The first, most pleasant, discovery was how easy it is to start using these tools. All of them have easy to use web-interfaces and provide extensive documentation. It took me just a couple of hours from opening a tool for the first time in my life to having my first automated test running. Of course, to use any of these tools to their full potential would take a much greater time investment. Using advanced features like adding code snippets, reusing steps, adding variables, setting up test data, managing test suites and execution requires more training and practice.
Codeless Script Generation (recording)
The next feature I explored is codeless test generation. This feature is implemented in all tools I tried with different levels of maturity. I found that Mabl is the most mature one. Recording the test using this tool was easy. It was able to handle some challenging navigation tasks, such as switching between different web-applications, finding and selecting checkboxes in dropdowns, and hovering.
Recording tests in TestCraft is slower and requires an extra step to record each action. This might be a showstopper if you plan to involve anyone outside the engineering team in test creation. The real advantage of test recording is that anybody can do it. You can ask the product team to record the user acceptance tests, ask the support team to record flows where end-users noticed issues, or even ask real users to record their actions. QA engineers can modify these recorded flows and add them to the regression test suite later. It should not require anything except hitting the record button and forgetting about it to be able to engage the non-technical part of the team or end-users in the test recording. If there is anything extra, chances are that they refuse to do it. Unfortunately, in TestCraft it is not possible. I also found that navigation between different modes (recording, execution, editing) is a bit confusing in this tool. On the other hand, I should mention that tests recorded in TestCraft were very stable from the get-go and required very little editing after recording.
Test recording in Testim (community version) was the least mature of any of the tools. I was able to record the tests easily, but they failed in multiple places when I tried to re-run them immediately after recording. Switching between different applications or selecting an item from dropdown required adding some delays, re-recording specific actions, and so on. I did not face these challenges with other tools.
Decrease of maintenance time
It is hard to estimate the advertised decrease in maintenance time during the evaluation period, especially if one has a limited number of automated tests (which is usually the case for Proof of Concept). There is also a need to have the same tests automated using traditional tools that are used in parallel to be able to compare time spent on maintenance of the same tests.
Tests I automated using the different AI-based tools were executed on multiple builds and releases. Every tool I tried handled insignificant changes in the UI, such as text, size, or other attributes, well. If the change was something that couldn’t be handled by a tool (for example the introduction of new elements, or a major redesign of UI) it was much easier and faster to fix an automated test case by re-recording steps than by introducing the same change using code.
