Mobile Quality Automation by Anyone from Anywhere
How to scale mobile quality automation
2020, What a year!?
The effects of COVID-19 are a game-changer all across the globe,
with significant impact on the technology sector in particular.
The lockdowns and quarantines caused an acceleration in remote working,
and even more importantly caused re-thinking of work methods and solutions.
This year taught us many things, but the main one is that reality is always changing.
For the sake of this blog, think about our reality as a game we play and our workplace as the playground.
The most important thing that we must ask ourselves while we are playing is how can we improve our game to better fit the surroundings?
As part of the Mobile SDK Quality Development team at AppsFlyer for the last 2 years, I can say that I’m constantly learning new things everyday since the playground is always changing.
So let’s start by defining the playground — the AppsFlyer SDK:
The AppsFlyer SDK is a library of code,
which the app developers implement in their code.
By implementing it, and following GDPR compliance and regulations, the developers will be able to use the SDK APIs, upon the end-user’s approval. This is effective to make sure that we report the user’s journey at the places that they want to use it.
For example, the end user just bought something within the app, or passed a level of a game in the app. It’s the type of information that the marketer wants to be familiar with.
Thanks to the SDK the marketer is able to measure and see the data of these events. So if we are talking about numbers,
the AppsFlyer SDK is in about 70% of the apps in the entire mobile market, and is being implemented in over 10B devices.
The SDK solution is developed for two native operating systems, one for Android and one for iOS, and on top of that we have 12 different wrappers (unity, react, flutter, etc.).
In my team, our main responsibility is to protect our SDK and make sure it serves our customers. We are doing it by taking care of the automated quality side of the AppsFlyer mobile SDK. It’s manifested by developing mobile test apps that implement the SDK APIs and are consumed by automation, writing our apps as the first app developers that implement the SDK. In order to cover the maximum functionality, we must give the SDK the proper environment to collect data, as in real life.
That is why we must use real devices to ensure all edge cases are being covered.
Once upon a time…
Our solution included a Mac mini, located in our office, that builds the apps that we develop.
While the tests are running, these apps are being installed over real devices.
Those real devices are running the scenario with a relevant test manager, let’s say Appium. Within a certain scenario, we send the data to AppsFlyer servers via a proxy so we store it all in a DB. Then automation can gather it, and make decisions on whether the test is failing or passing according to this data.
Now, think about the above process and imagine that we are running this entire solution in our own physical device lab with a hub for the devices.
Think about all the maintenance that we need in order to make it happen.
Having a local device farm increases the need to manage remotely a lab by API in device management layer , pull of devices , physical equipment, temperatures of devices and lab, the time consumption of integrating a new device for this solution each time, etc…
In addition to all of that, during 2020 we started to work from home (hybrid work model). As a result, we couldn’t be at the office to maintain our device lab. Not only did we discover that our complex solution was nearly impossible to maintain remotely,
It was also really hard to train new employees from a distance.
So, we got to the point where the whole team was running tests locally at home on local devices — there was no transparency and no test results could be shared. Even more importantly, all of our time was spent on maintenance. And the main question that arose from this situation was how could we scale our quality?
One of our main pains was to be able to run fast and deliver with high quality deliverables. So, how could we improve our game?
In order to answer, we needed to ask ourselves a few more questions:
- Accountability
Who is in charge of quality?
What would happen if all the SDK developers took accountability, could we make it better?
Could we make more people accountable for developing the tests and running scenarios? So yes, it means that the accountability for quality should be owned by everyone within the SDK scrum. - Impacting the feedback loop
Is running test scenarios over a physical device in our lab in the best interests of decreasing the feedback loop?
Since we are a hyper-growth company that recruits dozens of developers, eventually we will need to add more and more devices and more time for maintenance.
So the feedback loop time increases, because we are busy doing maintenance instead. - The human factor of the equation
Is it easy enough to show people around our solutions?
How easy would it be for a developer to use our solution, and how easy would it be for me to recruit and train someone on what we do within the team?
So we realized it was quite complicated and that we needed to change that.
In order to improve our game,
we started a new project — “By anyone from anywhere”
So, How can we develop our testing and execute them by anyone?
By Anyone
We took an existing solution of Java automation and we decided to switch how we develop our testing and engage developers to work in their own local environment. we switched it to two different integration layers.
One would be the Xcode and XCUItest, and the second would be the Android instrumentation tests.
That way each SDK developer will be able to take our testing app, and run the tests locally before the build stage. The developer will be able to write and run scenarios before the CI, even locally with their own device. That simple test gave us the ability to ask ourselves once more…
Who’s in charge of quality and who can consume it?
And now, both the SDK developers and quality developers can.
From Anywhere
So we understood that we needed to switch the current local device cabinet solution into a more global solution. Since we needed to use real devices in our tests, we couldn’t use emulators to achieve this part.
Consequently, we had to dig deeper to find a better solution than our local device lab, which would enable us to use our own private network and private settings.
We also needed a place that would hold all the results in an organized report, so we could share it.
Eventually we chose a device farm platform that can host our test suites and apps, and just trigger it with a Jenkins API.
As an AWS customer, we chose an AWS device farm, and we discovered that it’s not just easy to scale and run the tests, it also provides us with so many things that we didn’t have in the past.
For example, now we have Console log, Crashlytics, and many other tools that will help us debug the whole flow. Once we had it, it was pretty straight forward. The idea of making the former complex solution that easy and approachable, allows us to gain more contributors with less effort than ever.
This blog was written in order to create some impact.
So my question to you guys is: How can you improve your game?
