The Future of Software Testing: AI-Powered Test Case Generation
When we look around, it is impossible not to recognize the use of artificial intelligence (AI), nowadays, in every aspect of our lives. Especially with the rise of the popularity of large language processing models, such as ChatGPT, Copilot, and Gemini, the infamous, sci-fic-based question, “Are robots going to take over the world,” left its place to “Is AI going to take over the world?”. Even though, as humans, we cannot guarantee the answer to these questions, it is certain to say that AI started to change the way we execute our work and shape the future of software development. In the case of test engineering, how can AI impact the way of generating test cases? Let’s take a look at the possible ways that AI can guide test engineers to conduct test case generation.
To start, it would be beneficial to mention the basic concepts. A test case refers to the actions to decide if the software meets the expectations for the functionalities and requirements determined for the software [1]. Test case generation refers to creating these test cases. Artificial intelligence is defined as computer systems that perform tasks that normally require human intelligence to achieve [2]. Machine learning (ML) is a branch of AI that accepts vast amounts of data to learn, makes predictions, and generates algorithms that enlighten the unknown patterns behind the data [3]. Supervised learning, unsupervised learning, and reinforcement learning make up machine learning [3] (Figure 1). Computer vision is also another field of AI, in which the computer can retrieve information from visual inputs [4]. On the other hand, natural language processing (NLP) is also a concern of AI which provides computers with an understanding of human language and communication in the way that humans normally communicate [5].
Overall, in the topic, there can be more than one aspect of AI that may have an impact on test case generation. The contribution of machine learning is heavily involved in AI-powered test case generation. With supervised learning, the model can be trained by the past data. Hence, the model can predict the parts of the software that are likely to contain defects. This also yields optimal test coverage since the scenarios, which are determined during test case generation, are concerned more with the detected parts by supervised learning. Furthermore, supervised learning models can be trained to generate test cases from the historical data, minimizing the manual work done by the test engineers. On the other hand, unsupervised learning models can also uncover any hidden patterns for the placement of the defects in the software. These hidden patterns can guide the test engineers to also focus on the disregarded parts of the software to generate test cases. Lastly, reinforcement learning can figure out the most optimal sequences for test cases to increase defect detection and decrease the execution time at the same time. Also, the model may decide to prioritize specific tests, based on the likelihood of finding defects. Moreover, the model may be capable of adapting to the changes made in the software environment, making sure the test cases are still useful [7].
Utilizing computer vision is another aspect of AI that can contribute to test case generation. With image classification and object detection techniques, a model can recognize components on web pages, such as buttons, labels, and error messages, when trained (Figure 2). In the traditional approach, codes take the direct reference of elements, using identifiers (e.g. class name, ID). When computer vision is used, first the model treats the website as an image, scanning the overall view. Thus, it identifies components, based on its training. Unlike the traditional approach, the model can acknowledge the components, based on their appearance, without the need for identifiers. Eventually, the test code can work through the output of the location of the specific component by the model. This ML approach specifically brings benefits of easier adaptation to the updates in the project (e.g. changes in the user interface) since it does not depend on the detailed implementation, and it can detect the components without the need for the identifiers [8].
NLP, on the other hand, is another powerful approach to use in test case generation. An NLP model can analyze the software documentation that includes functionalities, requirements, bug reports, etc. Thus, it can learn how the desired software should act and what is needed to test the software. Furthermore, it can create optimal test case scenarios that cover the software documentation. On the other hand, the model can also learn from previous test flow sequences, generated by test engineers. Hence, it can yield similar test flows by itself, depending on the pattern of the data provided [8].
Overall, the involvement of AI in test case generation brings many benefits to the table. Using the mentioned AI techniques can reduce the manual effort required by the test engineers [9]. In this way, test engineers can focus on more critical parts of the test procedure. Furthermore, with the use of ML, error detection and code coverage can be optimized to make the test procedure more effective and efficient [9]. However, for future reference, we shouldn’t also forget that there are challenges for engineers when considering the integration of AI into test case generation. Data quality may be a concern, so if models are thought to be trained, engineers should ensure that the training data has high quality and is well labeled [9]. On the other hand, engineers also should be familiar with the trained models and know how the models decide during testing procedures [9]. Otherwise, it can be a lot harder to provide transparency and trust the models. However, despite the challenges, AI in test case generation is an undertaking concept that engineers and companies should not overlook this concept in the future.
References:
[1] “How to write test cases (with format & example),” BrowserStack, https://www.browserstack.com/guide/how-to-write-test-cases#:~:text=A%20test%20case%20refers%20to,necessary%20to%20verify%20a%20feature. (accessed Jul. 10, 2024).
[2] “Artificial Intelligence,” IEEE, https://globalpolicy.ieee.org/wp-content/uploads/2019/06/IEEE18029.pdf (accessed Jul. 10, 2024).
[3] “What is machine learning (ML)?,” IBM, https://www.ibm.com/topics/machine-learning (accessed Jul. 10, 2024).
[4] “What is Computer Vision?,” IBM, https://www.ibm.com/topics/computer-vision (accessed Jul. 10, 2024).
[5] “What is NLP (Natural Language Processing)?,” IBM, https://www.ibm.com/topics/natural-language-processing (accessed Jul. 10, 2024).
[6] R. Khadka, “Machine learning types #2,” Medium, https://towardsdatascience.com/machine-learning-types-2-c1291d4f04b1 (accessed Jul. 10, 2024).
[7] A. Ibrahim, “AI in Software Testing: Enhancing Speed and Accuracy,” Academia, https://www.researchgate.net/publication/381843281_AI_in_Software_Testing_Enhancing_Speed_and_Accuracy (accessed Jul. 2024).
[8] D. Santiago, T. M. King, and P. J. Clarke, AI-Driven Test Generation: Machines Learning from Human Testers.
[9] C. Deming, M. A. Khair, S. R. Mallipeddi, and A. Varghese, “Software testing in the era of AI: Leveraging machine learning and automation for efficient quality assurance,” Asian Journal of Applied Science and Engineering, vol. 10, no. 1, pp. 66–76, Dec. 2021. doi:10.18034/ajase.v10i1.88
