Delivering High-Quality Releases efficiently with App Automation

Consumer centricity has always been at the heart of everything that Airtel does. To keep up with the rapidly changing customer needs and ensure a seamless experience, it becomes important to put extra focus on effective Quality Assurance. Airtel Thanks App is a significant contributor in serving the scale of more than 350 Mn customers with an installed base of 100 Mn with 15 Mn DAU (Daily Active Users) which makes Mobile Application testing even more critical.

In this article, we will talk about how App automation is important in ensuring rapid deliveries and providing regular quality checks at various stages in the software process.

Key Challenges in Mobile Application Testing:

• Device Fragmentation Coverage: With Too many devices globally. Compatibility testing of the application on different versions across different OS is a must.
• Strict Timelines: Along with increasing User demands, Patchwork, bug fixes, and upgrades are other requirements that keep developer and testing teams on their toes. Automation makes life easier by reducing the bandwidth for regression use cases.
• Stability impact due to external dependencies and failures: Unexpected App Failures due to backend releases, minor releases, or infra failures. With daily scheduled automation and alerting in place, the turnaround time to identify and fix issues increases tremendously.
• Seamless User Experience w.r.t App Performance: App Health is another metric that causes greater User dissatisfaction. Automating the performance metrics like Launch Time, Page Rendering Time, and other Application Statistics helps to identify issues due to any code changes in Pre Production stage itself.

How did we proceed to solve these problems?

Automation testing has great practical usage and that is why it is increasingly becoming a must-have for projects as it solves all the above problems with the right framework and test coverage.

Key metrics while designing the App Automation Framework:

• Production Stability — 0 production bugs in user journeys and stable App Health metrics due to release changes. Daily scheduled runs to improve turnaround time to fix issues due to other failures.
• Faster Deliveries — 0% QA b/w for minor releases. Reduction in QA b/w by 70% for major App Releases.
• Automation as a Governance Platform for new features — A common platform across Airtel extended for different types of mobile applications leading to only specific manual reviews and interventions during release stability testing.

CORE CAPABILITIES:

• Platform compatibility — Support emulators and cloud but also go on to support real devices for the biggest platforms i.e. Android and iOS.
• Multiple mobile application support- Framework should work for all three major types of apps (native, web, and hybrid apps).
• Multi-threaded architecture- Framework should be able to run on multiple devices and parallel instances.
• App Health Metrics— App Performance testing support
• Reusability — Reusable code modules that allow you to conduct data-driven testing.

GOVERNANCE PLATFORM:

• Integration with CI/CD Jenkins Pipeline — This further includes Size Check validations, UT Coverage, and alerting on ANR/Crash Rate along with sanity checks.
• Integration with Test Data Framework to get Test Coverage for written test cases.
• Integration with Influx to review App health on Grafana dashboards.
• Integration of Backend APIs to identify Api failures and data validations.

Technical Specifications​:

Appium Architecture Diagram:

Architecture Workflow:

Appium is an open-source mobile automation tool that allows mobile app automation i.e. native, react native, and hybrid app. Appium architecture can be divided into four parts:

1. Appium Server: acts as a mediator between the mobile device and client library i.e. Automation scripts. Its job is to collect the data as commands from the client library and use it as a Mobile JSON wire protocol to pass it to the respective devices via bootstrap in the case of Android or web driver agents in the case of iOS devices.

2. Client Libraries: Client libraries are language-specific bindings that are used to interact with the Appium server. These libraries provide powerful APIs that allow one to write test scripts in their preferred programming languages, such as Java, Python, and JS, and in our case it is Python.

3. Mobile Devices: The mobile device is the platform on which the application under test (AUT) runs. It can be real devices on local device farms and cloud-like Browserstack. Mobile devices receive the command for the Appium server as every test command has capabilities (i.e platform name, automation name).

4. Automation Scripts: These scripts use client libraries to send commands to the Appium server, which executes them on the mobile device.

Simplified Framework Workflow:

How we defined Automation Coverage?

TEST COVERAGE

• Page Rendering Automation
• Functional test scenarios for User Journeys
• Performance Metrics Test — Cold and Warm Startup Time, Page Rendering Time, Janky Frames, CPU, GPU, and Battery Usage.

Tools Used:

Test Rail

Jira

DEVICE COVERAGE:

• Android and iOS are both platforms to be considered.
• Browser compatibility for mobile-web and web flows.
• High Traffic Devices — Low, Medium, and High end with varying Screen Density based on Analytics Data.
• App Versions with new releases and upgrades.

Tools Used:

Firebase Analytics

Browser Stack for Device Coverage

What we were able to achieve:

1. Automation Suite execution time for ~200 Test cases ranges between 25–30 minutes for Android and iOS.

2. Can run on different devices in a single run with ~25 parallel threads.

3. Common locator strategy for both platforms.

4. Error Logging and failure capture capabilities

• Identifying API Failures and duplicate API Calls via accessing network logs for BrowserStack session Id.
• ADB Errors and Crashes along with Backend API Responses captured in reports.

5. APP Performance Metrics were captured and integrated with Jenkins

• Janky Frames, CPU, GPU, Memory utilization, battery usage, and Page Rendering Time through adbDumpsys on Browser Stack
• App Launch Time includes Cold, Hot, and Warm Startup Time preferably through adb shell commands on Physical Device.

6. Integration with Jenkins pipeline

• App Release and Rollout Builds
• Daily scheduled runs for current and highest adoption App versions.

7. Integration with Test Rail to identify the Test Coverage for the automated modules.

8. Influx Dashboards were built based on Automation Test Results to identify the passing percentage and create alerts on high failure rates.

9. Script failures were reduced to 5% with the below solutions:

· Integration with Backend APIs for data validation to create generic scripts.

· Specific Test Data for Automation runs.

· Static and Dynamic Mocks for use cases that could not be tested on Production data.

· Test Data State check validations to ensure the test case is run on correct data and remove false failures

· Rerun Capability for Failed cases

Automation Reports:

Appium Report Link

Grafana Dashboard Link

Success Metrics:

• 60% of reduced QA bandwidth in major releases and tech stories.
• 100% of reduced QA bandwidth in minor releases.
• A total of ~500 Test cases have been automated till now for App journeys and ~60 bugs have been reported specifically through Automation including both Functional and Performance.
• No bugs in automated use cases for App Production Releases.

Coming up next:

• Integration with CI/CD Pipeline for App Releases
• Analytics Validation

Special Thanks: Platform Team at Airtel including Chetan Paliya, Yatin Sharma, and Himanshu Gulati for continuous support and guidance.