A cloud-based test automation architecture in AXA France
Abstract
The Test Guild of AXA France is composed with more than 200 test engineers, test leads and engineering managers. We are working on qualification of IT projects with a large variation of technologies behind. It is necessary to build a generic Test Automation Architecture, to cover different types of test, such as API Testing, Mobile Testing and UI Testing, also to provide a consistent way of test automation for test engineers.
In this article, we will discuss our Cloud-Based test automation architecture, how we have built it, and how automation is involved in the agile development process and integrated in the DevOps environment.
Introduction
Starting from 2016, AXA France has reorganized its IT department. By adopting Agile at Scale, Teams such as Developers, Testers and OPS engineers are working together in the same Tribe. Specialists of the same profession are organized in a Guild.
Test Guild is composed with more than 200 Test Engineers and Managers work in 20 tribes covering every major projects. Giving the size of the Guild, number of projects, and variation of technologies used, it is important to provide a consistent way for test automation, and to build a generic Test Automation Architecture covers the needs.
Test Activities
Test activities in AXA France start at the beginning of the development cycle.
At MMF (Minimal Marketable Feature) level System Test scenarios are identified and evaluated with Product Owner. These scenarios will be tested during system test, while critical scenarios will be covered by End-to-End test automation.
At US (User Story) level, during a 3-amigos session, Acceptance tests will be defined and being tested during a Pair-Test session.
User Stories will be refined to actions on each component, Test Engineers will ensure Component test using API Test tool (Web Service) if possible.
Test Engineers are involved in all 3 levels of manual test: API Level, Feature Level and System Level. Test Automation Engineers ensure API test automation and End-to-End automation, While Front-end Developers will ensure test automation on Features using Behavior Driven development (BDD) and Gherkin Scenarios.
To meet the need of these levels of test automation, we have built the following Test Automation Architecture based on Hybrid Cloud environment
Test Automation Architecture
Test Automation Architecture (TAA) developed by AXA France covers Test Execution Layer and Adaptation Layer, connected to Requirement Management system and Test Management system.
This generic architecture has 5 main parts:
Workstation, DevOps Environment, Test platforms (on Public or Private cloud), System Under Test and Test management
Workstation
Workstation is either local PC or a virtual machine with necessary tools installed. Test Automation Engineers will develop and test their Test Automation Solutions (TAS) locally, using software approved by the Test Guild:
● UFT Developer for UI testing, combined with the same development tools Visual Studio or InteliJ IDEA used by developer.
● ReadyAPI for API Testing.
● Selenium on BrowserStack for Mobile testing.
DevOps Environment
AXA France uses Azure DevOps to manage Repositories and CI/CD pipelines.
Once automation solution is developed and tested, it will be committed to the Git Repository, the same way developers are committing their codes. Git Repository holds different versions and branches aligned with the version of the system under test.
CI/CD pipeline defines the procedure how and when tests are executed. API tests are generally launched after the deployment to dev and staging environment. End-to-End tests are either triggered manually or launched daily to ensure the absence of regressions on test environments.
Three agent pools are established for API, Mobile and UI tests. Each agent pool contains one or more virtual machines hosted on test platforms.
Test platforms
The third part is where automated tests are executed. Test Agents (Virtual Machines) are connected to different agent pools for different type of tests. API test agents are hosted on private cloud while UI test agents are hosted on public cloud. The details of test platforms will be discussed in next chapter (Cloud based test platforms).
System under test
The fourth part is our system under test (SUT), the different test environments in a complexe ecosystem. They are protected by Firewall and Network Security groups. While using BrowserStack for mobile testing, we used its “Local testing” feature that establishes a temporary VPN like connection between device in the cloud, to the virtual machine to secure the SUT
Test Management
We use JIRA to store Epic, User Story and Defects, and XRAY to manage test repository, test execution and dashboard. All test results include automated test results are integrated into Xray.
Cloud based test platforms
At execution layer, we have 3 different test platforms where API, Mobile and UI testing are running. According to test type, Test Automation Engineers will choose an appropriate agent pool. The way of launching these 3 types of automated tests are the same.
API test platform
API test mainly targets middleware and backend web services. For security and maintainability reasons, we have installed these virtual machines on our private cloud. Testing tool ReadyAPI and other necessary components are installed on each server.
API Testing is not very CPU intensive, to maximize resource usage, 2 test agents are installed on each machine. This platform can run 4 API tests simultaneously. As a result, More than 200 API test projects are executed everyday.
Mobile test platform
As for mobile testing, we have adopted the solution BrowserStack for the variation of the available devices. Tests are executed on physical devices in BrowserStack’s data center, via virtual machines on our public cloud.
For security reasons, SUT is not exposed over internet. when testing applications hosted on AXA’s internal network, we use “local testing” feature. which temporarily opens a secured VPN-like connection between the device and virtual machine. With this feature, we can ensure that internal applications are not exposed to the Internet and device cloud, while the test automation can be run on a secured environment.
UI Test platform
System Test is generally based on User Interface. They are very time intensive. Secondarily, many test engineers adopted DevOps integration. As a result, there are many tests queued in the early morning but very few at on afternoon. It’s hard to determine how many servers are necessary to fit the need, in the same time do not exceed the budget on test infrastructure.
To address this problem, we have developed a solution based on Cloud Computing using Azure Lab Service. Cloud computing has two advantages which complied our need.
● Elasticity: The ability to Scale-Up and Scale-Down the power of the infrastructure according to the need.
● Pay-as-you-go: To pay only the resources used.
When the number of tests requests goes up and down during the day, we need to adjust dynamically the capacity of the platform, to meet the needs of test automation engineers and keep the infrastructure budget at a reasonable level.
Based on Azure Lab Service, our UI test platform has been built in 3 steps:
1. Creation of a Disk Image
First, build a disk image with all required software installed, configured, and tested. In our case, we have installed UFT Developer, development tools and different browsers. This image can be seen as a template, it will be used to create other test agents.
The disk image needs to be recreated, if Operating System, Test tool or browsers are updated. To reduce the complexity of maintenance, we collaborated with our IT Support team to automatize the installation of software, and creation of disk image via scripts.
2. Automated VM Creation
The second step is to automatize the creation of VM based on previously created disk image. If more test agents are needed, this automated script can create new virtual machines and add these new virtual machines into the agent pool.
This script is implemented using Azure Lab Service (aka DevTest Lab).
3. Monitoring and Auto-Scaling
The third step is to develop an independent service to monitor how many tests are queued and how many machines are needed, then automatically adjust VMs in the pool.
The auto-scaling service is based on following parameters:
● The number of enqueued automated tests.
● The number of agents available in the pool.
● Prediction of “How long it will take for current test to finish” based on historical data.
With these parameters, the service will find out the best solution for time and budget. For example: At early morning, as many tests are queued, the service will startup all available machines. If not enough, Create additional machines to ensure tests can be run as quick as possible. When one or more agents are idle and while no more tasks to run, the service will shut them down.
Another advantage of the centralized monitoring is that, the usage of platform is transparent. We know how many and when tests and launched. The data can be served to improve the solution or to predict how many licenses are needed in the future.
With the scripts and Auto-Scaling service, we now have an automatic and scalable solution for UI testing, of which we can run any number of tests simultaneously at any time. The solution is highly budget friendly.
Common Tools
Apart from those test platforms, we have implemented some DevOps tasks for test automation engineers:
● Xray Import Task: This DevOps task will be used to integrate test result into the Jira/Xray test projects. Manual tests and automated tests can be managed together.
● Send Mail Task: This task will send a test result resume by mail. When the test is finished, Test Automation Engineer or anyone listed, will receive a resume by mail.
● Test Data Downloading Task: As many automation projects apply “Data-Driven Test” method, this task will retrieve test data from external source in real-time. Test data can be managed independently by Non-technical testers.
● Test Report Upload task: We do not import detailed report (with attachments and screenshots) to Xray (for Jira/Xray performance concerns). This task will be used to achieve test report to the location indicated by automation engineer.
These tools simplified some common tasks for automation engineers, they can focus more on essential tasks of test automation itself.
Conclusion
This architecture covers the need of API, Mobile and UI test automation in AXA France, integrated with the Management tools used by the company. For test automation engineers, any kind of automated test can be run in a DevOps environment easily, consistently, and seamlessly.
As we use Azure as public cloud provider, the solution is built on Azure Lab Service (DevTest Lab API). But all Cloud computing suppliers provide resource management API, similar solutions can be considered based on cloud service provider used by your organization.
Authors
This article is co-written by Huaxing YUAN and Jean-Prince DOTOU-SEGLA.
