Insider’s Mobile App Test Automation Journey

In 2020, Insider assembled a new Mobile Apps team with an eye towards redesigning and building a new Insider application from scratch. As part of the team since its launch, I wanted to share the experience of how we improved our mobile app automation strategy over time. It’s been a learning experience for me; despite my extensive background in test automation for the web, it turns out that the process is quite different for mobile apps.

I joined Insider as part of the Mobile App team. At the time, we were maintaining two separate codebases, one for each platform, iOS and Android. Each required a different set of tools that had their own nuances and intricacies. I had a chance to explore and write some tests using XCUITest and Espresso (for iOS and Android). Covering even the most basic flows was difficult. In the end, I was able to put together a small suite of tests that covered some basic functionality. The engineers on the team would run these tests locally to smoke test the app while working on changes. It was a less than ideal situation.

I decided a better solution was needed. The current version of the app is built using React Native, which led me to come across Detox, a testing and automation library for mobile apps. The nice thing about Detox is that it exists to test only React Native applications.

Soon after, I started experimenting with the tool. I found a way to build the app for use with Detox and wrote some preliminary tests. We focused on writing tests for iOS, since most of our user base is on that platform. Detox also had limited Android support at the time.

It was important for my team and I to have tests that we write run on the creation of pull requests. I started exploring ways to run Detox tests with GitHub Actions. Since we’d need to build the iOS version of the app for testing, a Mac virtual machine was required to build and run our tests. Github Actions hosts Mac runners to use with their CI tool. Unfortunately, building in CI using these runners is a slow process.

The job to build the app and run tests against it often took close to 40 minutes with the Github hosted runner. Github charges 10x more per minute of use for this runner compared with their Linux runner. This, coupled with the poor performance, meant we had to find another option. At this point, the tests we had weren’t very valuable for the team due to how long it took to get feedback from them.

While looking into solutions for the performance issues we were facing, we learned that it was possible to set up physical machines as Github Actions self-hosted runners. Around the same time, the IT team at Insider was working towards building a closer relationship with the rest of the Tech team. This was a great opportunity for the two teams to work together. The IT team gave us access to a physical Mac Mini that we were able to configure remotely to use as a self-hosted runner.

After solving some challenges related to dependencies, we were able to run our first job against our self-hosted runner. The dependency installation step execution time was drastically reduced. The most stunning difference involved the time it took to build the app. The build step went from around 30 minutes to under 10 minutes to complete. The entire job took about 12 minutes to complete, including dependency installation, linting, jest unit tests, building the app, and running our Detox tests. With the performance increase, we were able to consistently run all the tests on pull requests and provide feedback in a reasonable amount of time.

With the performance improvements in mind, the engineers on the team began to take the Detox tests more seriously and to take more interest in them. Today, with the entire team’s involvement, we have a test suite with functional and visual testing. With it, we’re covering many of the app’s critical user pathways with plans to cover even more. We also got more machines to use as runners!

More recently, the team worked on utilizing mocked data to populate certain parts of the app. This allowed us to isolate components that would otherwise be difficult or impossible to test using automation. The team has also gotten into the habit of making the app more testable by habitually adding Test IDs to components for easier targeting with our tests.

As we progress, we have some areas of improvement in mind. We plan to support testing of the Android version of the app. We will also explore ways to handle automating more complex Subscription and Payment flows. Like the app itself, the state of the team’s test automation has been improving and advancing consistently with time. I’m looking forward to the future iterations of the app and the tools we utilize to test it.