QA engineers, functional and UI testing
The evolution of apps QA at Azimo
This is the second in a series of blog posts in which we outline our multiple years’ experience with our Android app testing at Azimo. Most of the principles, goals, and achievements also apply to our iOS app.
Table of content
- The evolution of apps QA, first days, and unit testing
- (This post) — QA engineers, functional and UI testing
- Custom tooling for testing stack management
- Removing and subtracting tests is part of development too
- QA testing in the cloud
QA engineers in the team
Let’s recall our original problem described in the first blog post in the series. It wasn’t about missing unit tests in the first place but about the QA team not being available immediately when we needed them. Why then we created hundreds of unit tests instead of hiring a dedicated QA engineer?
Even if we could do this, we wouldn’t be able to keep him or her busy most of the time. Our app was released no often than once a month. It wasn’t only because of limited access to the QA team but also the questionable quality of our code and bugs and almost no process automation.
Unit tests changed this situation. Writing them improved our codebase quality and overall app’s architecture. With the approach “if something is hard to test, maybe it’s a bug,” we modularized our project, broke down colossal classes into simpler ones, introduced dependency injection, and abstracted SDKs away from our app’s logic.
These changes allowed us to take a step forward and open a full-time QA engineer’s role in the mobile development team. Cleaner code meant changes delivered in smaller chunks, which meant fewer bugs. Better modularity and abstractions gave us the possibility of writing functional and UI tests. We could redefine the QA tester role to QA engineer and focus on a new ambitious goal — 2 weeks release train.
Functional / UI testing & automations
The following months of QA engineer’s work were about two things:
- Testing the changes coming with new releases,
- Automating these processes as much as possible.
One of the goals back then was “80% of QA manual work is automated”. We didn’t want to spend more than one day per week doing manual tests.
Testing tech stack
We built our testing tech stack on top of publicly available tools. We used Espresso as a testing framework and Fastlane for building pipelines and automation. At the beginning, they were more than enough for our needs. Later, when we identified some gaps, we also published some of the open-source projects extending these frameworks:
- ConditionWatcher — the solution that solved some of our problems with asynchronous operations and the way Espresso works with them. To read more about our challenges with its IdlingResources, look at our blog post: Wait for it… IdlingResource and ConditionWatcher. We cannot be happier with this tool being described in the book: Android Espresso Revealed. We are honored 🙏.
- Fastlane Plug-in for managing Android emulators — thanks to it, we could move AVDs management directly into Fastlane pipelines. More about it in our blog post: Managing Android Virtual Devices during test session.
Like with unit tests, we wouldn’t start creating functional and UI tests without a good reason. It’s not (fully 🙃) about being sexy in the engineering ecosystem. There were some challenges we needed to tackle. Here are the most important reasons behind why we started building functional and UI tests.
Unit tests aren’t enough
Somewhere around 50–60% of code coverage, we noticed that more unit tests don’t necessarily make the quality of our product better. While there were fewer close-to-code bugs, our QA engineers kept finding product issues related to 3rd party integrations, API connection, or SDK (e.g., data not being correctly passed between screens).
In the perfect world, appropriate abstractions, API contract testing, and all 3rd parties being covered well tested would solve these problems. But we don’t live in one.
Side note on abstractions — sometimes, it’s much easier to write UI tests within days instead of spending weeks building another abstraction over things like navigation, files/camera management, permissions, etc.
Fragmented environments
There are numerous screen sizes, each manufacturer introduces its changes into Android OS, year over year, we get new system versions, and there are high-end and low-end devices. Unit tests won’t tell you about intensive garbage collection, not implemented scrolling container, or missing backward compatibility for Android 4.0.
Test things in the easiest place to test them
Building a start-up product is a constant balancing between speed and quality. Sometimes you must take shortcuts to survive. One of the shortcuts, in our case, was testing some of the backend features through client-side platforms. And if you need to do it manually, why not write automated UI tests doing it for you.
Preparation for tech excellence
In some situations, existing solutions are so complex that blindly rewriting them from scratch would bring many bugs. But to scale up, we need to keep refactoring them. If it’s too hard to write unit or functional tests in the first place, you can always try UI tests.
For example, before we recreated the registration process for 20+ countries in a fully testable, elegant solution, we checked a registration process for every single country via UI tests. And thanks to them, after we finally unified this logic, we were sure that we covered all of the edge cases in our new solution.
We spent long months adding more functional and UI tests to our code for the above reasons. Our approach was to cover functionalities starting from the most important ones:
- Registration and login,
- Transaction, payment and price checking,
- Everything else: referrals program, documents upload, settings and others.
Our biggest win so far — before functional and UI tests, QA engineers needed about one full day to test our product manually. With automation, we needed the same amount of time, but we could cover significantly more cases. And our machines were never bored and never tired of doing that (e.g., mentioned tests for each of 20+ countries).
Side note: we wrote a separate piece about the benefits of testing automation here: Automated testing will set your engineering team free.
But these benefits didn’t come with no price. With so many things being covered, we ended up with hundreds of tests that took many hours to run. It brought us new challenges, which we will cover in the next blog posts. Stay tuned!
Towards financial services available to all
We’re working throughout the company to create faster, cheaper, and more available financial services all over the world, and here are some of the techniques that we’re utilizing. There’s still a long way ahead of us, and if you’d like to be part of that journey, check out our careers page.
