Scriptless GUI test automation at ING
Could we have some “monkey” to test our GUI applications before release?
Introduction
Traditionally, graphical user interface (GUI) test automation is implemented with test scripts, either recorded with capture and replay tools, or written with scripting tools. These test scripts consist of test steps that are GUI interactions, and test oracles usually in form of assertions.
For some years now, a scriptless testing approach has been emerging. The idea is not to (entirely) replace scripted testing, but to complement it. It does so by reducing the effort to create test scripts and covering paths outside the happy user scenarios, concentrating on robustness testing instead of functional testing. So far, the approach has been applied successfully in various industry domains. In this blog, we present the scriptless GUI testing tool TESTAR (we also refer it as monkey testing for GUI) and show how it is being deployed and evaluated at ING to test web and mobile applications.
Our work is done within the context of the IVVES project, an ITEA3 project number: 18022 from 2019–2022, whose goal is to develop approaches to industrial-grade V&V of systems containing Artificial Intelligence (AI) as well as V&V using AI. The project covers major industrial domains in The Netherlands. One of the projects’ challenges is developing scriptless intelligent automated testing tools for event-based systems.
The goal of the evaluation is to measure test effectiveness (with state coverage and number of detected failures), test efficiency (time needed to prepare, run and analyze the result of the automated tests) and, if possible, over various releases of the web applications, compare maintenance effort of test automation artefacts of scriptless versus scripted testing.
TESTAR
TESTAR is an open source tool that implements a scriptless approach for automated test generation for event-based systems. In scriptless testing, the sequence of actions to test the system under test (SUT) is generated dynamically during the test execution, based on the observed state of the SUT. The underlying principles are very simple: generate test sequences of (state, action)-pairs by starting up the SUT in its initial state and continuously selecting an action to bring the SUT into another state. The action selection characterises the fundamental challenge of intelligent systems: what to do next. Currently, besides simple random action selection, TESTAR offers prioritised selection and AI-based selection based on Q-learning. Moreover, it is being researched how genetic algorithms can evolve the best action selection strategies.
Scriptless GUI test automation at ING
ING uses CI/CD pipelines to automate steps in the software delivery process, such as initiating code builds, running automated tests, and deploying to a staging or production environment. Automated pipelines remove manual errors, provide standardised development feedback loops and enable fast product iterations. However, before release of the application, manual GUI-based testing is still needed on a release build. Within ING, this type of test session is done with a manual regression test. The goal of such a test session is to find any remaining failures in the app after completion of all pre-release and integration tests.
To reduce manual work on GUI-based testing and find failures in ING Web and Mobile Applications early in the development process, ING wants to apply smart monkey testing for our applications using TESTAR. The figure shows an overview of the architecture of our demo. We will distinguish between 3 parts during the demo:
- TESTAR (Generic), i.e. testing an ING application with TESTAR out-of-box
- ING specific extensions that had to be developed (or have to be in the future), to cope with testing specific characteristics of ING applications.
- ING app specific extension
The goal is to improve TESTAR’s ability to monkey-test ING applications. Here we would like to present our current progress in IVVES with two demos of ING use cases: ING Mortgage Calculator and Covid Loan Application Flow.
Demo
ING Mortgage Calculator Demo
The video below shows how TESTAR tests ING Mortgage calculator. The green dots indicate all clickable buttons on the current screen (the left part is configured to be ignored). The red dot indicates the next step which the “monkey” will take.
Covid Loan Application Flow
The video below shows a more complicated use case, where the “monkey” is testing the ING application. In this case, it can go though different scenarios automatically and generate testing report at the end.
About our team : Software Engineering and Automation (SEA)
We are IT innovation chapter at ING OmniChannel. In our team, we focus on different innovation topics on software engineering and automation.
If you have any question on our work, please feel free to contact us via: sea@ing.com.
Acknowledgements
This project is partially funded by the EU ITEA3 Industrial-grade Verification and Validation of Evolving Systems (IVVES) project, under grant agreement “ITEA2019-18022-IVVES”. We also thank Pekka Aho, Olivia Rodriguez Valdes, Tanja E.J. Vos from Open Universiteit, The Netherlands; and Fernando Ricos Pastor from Universidad Politécnica de Valencia, Spain for our collaboration inside IVVES.
