Are AI approaches useful only for UI testing?

Recently, there has been a lot of buzz that application of AI technology is going to make software testing more robust, and easier. There already are several startups informing these predictions and making these predictions a reality. However, most of the narratives, whether intentional or not, are highly UI-centric. Many of these are driven by the pain of creating robust UI test scripts and maintaining them whenever UI changes. Examples include prediction of trends from experts such as Joe Colantonio, Lubos Parobek, Mark Buenen, Do Nguyen among others. I agree with many of the predictions from these experts.

If, however, I were to choose the area of testing where we want to first see significant improvements from using AI techniques, it would be functional API testing. First, as per the popular testing pyramid guidelines, the number of functional API tests should vastly outnumber the UI tests. AI techniques should be able to help with test case enumeration, scripting as well as test script maintenance — reducing the overall engineering effort significantly. Second, the usage of APIs is increasing rapidly due to the adoption of microservice architectures and intelligent cloud services. Given these drivers, and the increasing ability to gather and apply AI over operational data from applications (e.g., Splunk), I think functional and performance testing of APIs will see a fundamental re-think. Let’s take some of the predictions made by several experts in the test automation space and discuss how they apply to the backend API test.

Record and replay will see a resurgence

Record and replay approaches record user actions and outcomes on the reference version. When a new version is available, the same set of actions would be replayed, and the observed outcomes compared with the ones recorded. Record and replay approaches had received a lot of attention in the past for testing UI. The approach was exciting because it was very easy for testers and product managers to record a specific user activity and create a test. However, the comparison between expected and actual was always fragile due to changes in the UI code and layout. Recently, several startups are developing robust comparison techniques to handle changes in the UI gracefully using AI techniques.

In contrast to the functional testing application for UI discussed above, record and replay for API testing has been mostly focused on performance testing. I expect that the following drivers and enablers will result in an emergence of smart, highly automated approaches learning from recorded data for functional API testing.

• Standardization of inter-service interactions on frameworks such as Rest, GraphQL, and gRPC. It is now possible to define the contracts for interactions between services and develop automatic validation.
• With the increased focus and ability to gather and learn from data including from the traffic between services and from services to backend data layer, we will see a lot of intelligent techniques developed for robust and flexible comparison of expected and actual API responses.

Self-healing test scripts

Self-healing has been highlighted in the context of UI test scripts where it is easy to specify the desired goal in terms of target content shown in the UI. With an alternative specification of the goal, test script healing automatically updates test scripts to make them work even if the UI changes. Therefore, engineers do not have to invest effort to manually update the test scripts.

In the context of backend API tests, the notion of self-healing is more subtle because the comparison involves two aspects which are often conflated in the same test script.

• Schema comparison: Certain fields in the schema are required in the response. The test scripts must ensure the presence of these important fields besides highlighting the absence or presence of non-critical fields. In addition, just like the application UI, the schema evolves over time, and the testing process needs to be resilient to the evolution of the product.
• Data comparison: Whether the expected and actual values of the corresponding fields match for the responses.

Changes in APIs can modify either schema or data or both. Therefore, it involves a very different set of challenges than automatically healing UI test scripts. However, with already known machine learning techniques for schema and data mapping and static code analysis, we should be able to automatically heal API test script changes. Therefore, I believe that self-healing of API functional tests is set for significant innovation.

Automatic test case enumeration

Currently, test cases are handcrafted by product managers and engineers. Due to the limitations in understanding and visibility of the application use scenarios and the lack of people’s time these handcrafted test cases only provide limited coverage. Data-driven ML approaches are applicable for automatically creating comprehensive sets of test cases by learning from application usage logs. Even though no one has yet tackled this problem, we believe this use case will be solved for testing backend API efficiently and effectively.

Who will benefit from such developments?

As discussed earlier, the usage of APIs is growing significantly due to several reasons. Therefore, any team developing an enterprise software application will benefit from applying AI-driven techniques to functional API testing. We are developing such a solution at Mesh Dynamics. There are other related problems in functional testing of enterprise software applications such as functional UI testing, test data creation and inability to apply canarying that are also ripe for innovative applications of AI/ML technologies. As these technologies evolve, teams adopting them should be able to achieve both higher quality and higher velocity at a significantly lower human effort.