Amplifying Testing Efficiency: Pocket FM’s Native App Automation Odyssey

Pocket FM adopts automation across the funnel for quality assurance

Written by Ritul Mishra

In a world that is always connected — thanks to smartphones and affordable mobile data — we are always using one App or another. Needless to say, we want each of these Apps to be reliable, functional and secure. That’s because we want our regular-use things to be convenient, and we definitely want our personal information — something that we are constantly sharing with these App companies — to stay in safe hands.

The onus lies on the App owners. For that, companies need to ensure these Apps are secure, easy to use and equipped with all desired functionalities before they go live for public use. That’s specifically the reason why every company engaged in developing Apps and other software put extra emphasis on a thorough testing process before an App or a software is made available for public use. But, the testing does not stop here, it continues throughout the lifecycle of every App and software to ensure it is bug-free, secured and up-to-date.

For testing, most of the companies use the standard industry methods, and some add some additional testing mechanisms of their own to ensure the app or the software is bug free and secured to the core.

At Pocket FM, which is a non-music audio entertainment company, we follow the Application Automation Framework to ensure efficiency and security that the App requires. Pocket FM is designed for creators and listeners, and our testing mechanism needed to ensure the convenience of our target audience. Our development team needs a framework suitable for testing the user-centric App, accelerated development ensuring quality and seamless collaboration.

To be sure, testing an App or a software isn’t only about checking the codes. Good testing mechanisms need to ensure a lot more. It needs to include insights from real-time data and analytics, performance metrics, drive innovation and enhance experiences of the end user, or listeners in case of our Pocket FM App.

In this blog, we would like to outline the testing process that Pocket FM quality analysis and assurance, or the QA, team follows.

Challenges:

Starting from conceptualisation — new App or a feature, to selecting what needs to be there and what not, to designing the App or a feature that people would fall in love with, to writing the code, every step comes with its own challenges. Testing is no different. Rather, testing comes with a lot more challenges.

If one has to find the bad guys in the software development industry, it would probably be the QA team. That’s because it’s only the QA team that has the power to make everything look bad. It is the sole responsibility of the QA team to ensure that the APP, or a software, goes out as desired, reliable, convenient, secure and lovable. At the same time, if anything goes wrong, it is the QA team that would be held responsible.

At Pocket FM, the QA team follows Application Automation Framework for testing both Android and iOS Apps. However, implementation of the automation framework was not without challenges. Here’s a few:

1. Platform Diversity: Managing diverse platform-specific configurations and dependencies can be intricate. (Application for android and iOS for India and US were major objectives)
2. Cloud Device Farm Integration: Integrating with various cloud device farms demands navigating diverse APIs, communication obstacles, and provisioning a multitude of virtual or real devices with varying OS, version, and configuration requirements
3. Test Data and Script Management: Efficiently managing varied test data and scripts across platforms and test cases can be complex. Different locales have different onboarding steps and parameters. Naturally, test datasets are usually different for each
4. Version Control: Ensuring consistent version control and synchronisation between app builds and automation scripts is crucial
5. Scalability: As the app evolves, the framework must scale to accommodate larger test suites and different app versions
6. Execution Time: With the increase in application features, tests will increase hence resulting in more execution time

Solution:

Every challenge needs to be tackled in a manner that is conducive. One of the things that our QA team adopted was a comprehensive automation pipeline using Jenkins. This pipeline was designed to encompass three crucial phases — App building, cloud-based device farm deployment, and execution of automation suites. The end goal was to establish a robust testing process that leverages Jenkins’ capabilities and integrations effectively.

Here’s a list of a few things that we did to address the challenges:

1. Platform-specific Build Configurations: Creating separate downstream jobs for Android and iOS builds enabled platform-specific adjustments
2. Jenkins Plugins and API Integration: Leveraging Jenkins Browserstack plugin and custom shell scripts facilitated seamless interactions with cloud device farms
3. Centralised Test Data Management: Centralised storage and management of test data and scripts streamlined maintenance
4. Version Tagging: Incorporating version numbers into app builds and test scripts ensuring consistent compatibility
5. Modular Framework Design: Implementing a modular framework architecture that allows scalability and adaptability
6. Test Optimisation: Optimising the automation flow and removing the repetitive steps to ensure time for execution of suite is optimal

Framework Lifecycle:

The automation pipeline spanned multiple stages, including:

1. App Building: Creating downstream jobs for Android and iOS builds, configuring parameters, triggering build creation jobs based on platform, and passing essential parameters
2. Build Upload: Using a downstream job for uploading builds, utilising Jenkins plugins for Browserstack and shell scripts for other platforms, and ensuring secure API interaction
3. Automation Suite Execution: Fetching latest Build IDs, configuring desired capabilities, executing automation test suite jobs, and validating outcomes

Framework Level Utility:

The automation framework’s utility was multifold:

1. Efficiency: The automation pipeline streamlined processes, eliminating manual interventions and reducing testing cycle times
2. Scalability: It provided a scalable approach to handle diverse platforms and multiple build versions
3. Reliability: Automation ensured consistent, repeatable testing scenarios, reducing the likelihood of human errors
4. Reporting: Collect and analyse test results, generate insightful reports

Adding New Test Cases:

Integrating new test cases into the framework involves:

1. Adding the new test in the test class

2. Reusing the existing codes for flows which are repetitive and going through the abstract business logic layer where we will be writing logics of Android, iOS test flow

3. Calling the respective screen class which are needed for the flows in the logic layer

4. Adding the screen methods to do the screen actions(like click(), getText(), sendKeys(), scroll() etc)

1. Test and Push the code to the branch
2. Trigger the automation suite from Jenkins to check the report along with older test cases

Tools Utilised:

The QA team at Pocket FM used a range of tools to ensure optimal automation. Here’s a few of them:

• Jenkins: For automation pipeline across the chain
• Jenkins Plugins: The Jenkins Browserstack plugin was essential for direct-built uploads to Browserstack’s cloud device farm
• API Interactions: Jenkins’ shell script to interact with cloud device farm APIs for platforms beyond Browser stack
• Appium: For UI interactions during automation
• Allure: For report generation and graphical presentation of report data with history of Pass/Fail test cases
• TestNG: For automated testing and test configuration in Java applications
• Slack: For getting notifications of automation suite execution results along with report links and screenshots

Continuous Integration & Continuous Testing (CI/CT):

The QA team at Pocket FM also leveraged CI/CT practices through Jenkins:

1. Continuous Integration: Changes triggered automated builds and tests in response to new code commits
2. Continuous Testing: Automation pipeline ensured continuous testing, validating the app’s functionality at each stage

Conclusion:

The successful implementation of the Jenkins-based automation pipeline has helped the QA team at Pocket FM to overcome challenges and develop efficient testing processes. The framework’s division into three coherent phases further ensured smooth execution, efficient uploads to device farms, and reliable testing. The synergy of tools, Jenkins’ adaptability, and robust framework design have collectively drove Pocket FM’s commitment to delivering the non-music audio entertainment app that is bug-free and lovable.

Future Roadmap:

QA is a continuous process and it cohabits with the App throughout the entire lifecycle. In the foreseeable future, the QA team at Pocket FM plans to implement a few more things aiming to make the App better every day. Here’s a few things that are in the pipeline:

1. Expand Platform Coverage: Extend automation to cover additional platforms, widening their app’s reach
2. Enhance Reporting: Develop insightful reporting mechanisms to derive actionable insights from test outcomes
3. Integration with DevOps: Integrate the automation pipeline seamlessly with their DevOps practices for a holistic software delivery approach
4. Self-healing of suite failure with AI: Usage of AI for correcting the code which are not genuine failures and are false alerts
5. App performance monitoring: Monitoring battery data, network data and frame rates to get the performance matrices of the application