Quality assurance for mobile development at hh.ru
I don’t know what developers talk about at conferences, but QA engineers talk only about processes. How testing is organised, how many autotests there are, who and when writes them, where they are run, how quality is ensured at all stages of development? My article today is just about this — about how we build quality testing in hh. There will be a little bit of theory and a lot of practice. Let’s get going!
Who they are: a tester and QA
First of all, it is worth postulating the following: quality in hh is not the area of responsibility of the tester alone. Quality is everyone’s responsibility: testers, developers, designers, and product managers. We try to deal with this at each stage of product development. We want to begin with the ancient debate about testing. Now, we can distinguish two terms: testing and quality assurance, which are often used to refer to the same thing.
Testing is a verification of the correspondence between the real and expected software behaviour. Quality Assurance, aka QA, which is an acronym that is often used, is a preventive process aimed at ensuring that all necessary techniques, procedures, standards and methodologies are followed in the product development process and provide a fault-free result.
If you don’t want to be too nerdy, it would be easier to say this: testing is one of the quality assurance stages. At hh, testers are quality assurance engineers; they influence the entire development process, not just looking for bugs. Further, in the article, I will use the terms “tester” and “QA” as synonyms in order to avoid unnecessary nerdiness.
So we came to the conclusion that QA is not just about testing. If we’re talking about QA engineers, their job is not just to test all the buttons in a new feature and test them again before release to make sure nothing falls off. QA is involved in all stages of product development. Our goal is to release an excellent quality product in a reasonable amount of time. We try to ensure quality at every stage of development. We monitor the quality of code, interfaces, design, and the final quality of the entire resulting product. QA can improve any of these processes.
Feature development stages
Let me briefly remind you how we write code: what we had three years ago and what we are up to now. Two or three years ago, we were working on so-called trunk-based development. We weren’t happy with that, and over time we switched to GitHub Flow. I already told you about how we improved the whole process and how the transition was made in a previous article, so I won’t dwell on that now. Let’s move on to how the entire feature development process is organised.
The entire process can be divided into several stages:
- Idea;
- Demo-design;
- Technical elaboration, decomposition, and evaluation of tasks;
- Development;
- Testing;
- Final release — putting a new feature into production.
And now let’s dwell on each of the product development stages.
Feature idea
At the idea stage, product people generate a hypothesis, come up with a new feature or functionality, discuss the idea with the designer, and together they develop the initial layout. At this stage, ideas are never too rigidly fixed. There is always an opportunity to change something, add more detail to layouts, and think through user scenarios more thoroughly. We don’t concrete all the requirements at this stage, so that we can discuss them with the team later, and someone will suggest something else, maybe. Or we’ll realise that in this form the feature won’t work.
Design demo
When the idea stage is over, it is followed by the design demo. At this stage, the whole team is directly involved — two iOS developers, two Android developers, and a QA engineer. This is how a typical dreamteam in the mobile direction works, and we have five of them. Three of them in the job seeker branch, one in the employer branch, and one in the platform branch. Other than that, the hh team is not just a batch of programmers, but a set of different competency areas, each of which is important. Each developer isn’t just cool at writing code, they can also be great at design systems or writing a network layer.
At the design demo stage, the entire team and sometimes other interested parties take part. These are product managers from other areas, testers, and so on. The testers definitely participate at the stage of presenting the layout. This is because they usually know this product better than anyone else in the team. They can highlight weak points, ill-conceived scenarios, and interactions of the new feature with existing ones.
Our feature development process is always iterative: if something goes wrong after the first discussion of a feature, the designer and product manager simply go back to work on it. They may rework the layout a bit or rethink the user scenario a bit. After that, we meet again to discuss the new layout. If everyone is happy with it, then the layout goes directly into development. If not, we can redo it and discuss it again.
Engaging testers and the entire development team at the earliest stages is very beneficial for the process and the business as a whole. There is a simple rule: the earlier a fault or a problem is detected, the cheaper it will be to fix it. If the fault is detected at the stage of layouts in Figma, or within an idea at all, then it costs nothing to simply redraw or rethink it. But if the bug is detected directly at release or even after development, it is much more expensive to fix it.
Of course, in the era of cascade development, it was much harder to fix a bug in production. When releases were once a year and there were no online updates, if you had already released a bug in the production, it was very difficult to fix it there without problems. In the era of iterative development and agile approaches, where we do releases on the site several times a day and mobile app releases once a week, it’s much easier to fix a bug from the production.
But the problems still remain. If the bug got in the production and was in the paid functionality, which because of the bug does not work, then we need at least return the money to the users who paid for it. And in the worst case, we can also face troubles and unpleasant reputational risks.
Technical elaboration, decomposition, and evaluation
The next stage is the technical elaboration of the feature, its decomposition and evaluation. Developers and technical specialists, if necessary, take part in this stage. The participation of QA engineers at this stage is mandatory. At this stage developers can discuss the implementation of a particular feature: what the problems can occur, what will have to be changed in the existing functionality, and maybe even something will have to be refactored.
The tester listens carefully to what developers are saying, discusses, and can already plan his testing around potential problems. For example, if they hear that in order to develop a feature they will have to fix some element of the design system, they will probably also have to fix it in other areas where it is invoked.
At this stage, the tester can already estimate approximately how much time he will need for testing. We never strictly record this estimate from the tester, but we need it for further planning of the team’s workload.
Development
The next stage is development itself. In many companies, at this point, testers leave to write some test documentation, test cases, or to update something. We, on the other hand, write autotests intensively at this moment. But not for the new functionality, but for the one that already exists, has undergone some A/B-experiments and so on. There is always something that is not covered by tests and urgently needs them. We may also have unstable tests or tests that need to be updated. These can be dealt with.
But we don’t always wait for the full development cycle of a particular feature. The tester can take over a build somewhere in the middle of the process, if there is a partially working functionality ready, and see how it is implemented. After all, even at this stage we may already encounter problems, and the sooner we notice them, the better.
Testing
As you can easily guess, at this stage the testing of the developed feature takes place. The main way of testing in hh is exploratory testing. This is a type of testing that does not require writing test cases, but implies that each subsequent test is selected based on the previous ones.
This does not mean that the tester randomly presses buttons and sees what happens. Unfortunately, no. This method requires a great deal of tester’s experience and a very good knowledge of the product. The main advantages of exploratory testing that we see for ourselves are:
- Doesn’t require support and writing test cases;
- We are not subject to the so-called pesticide paradox;
- Increased testing speed.
What is the pesticide paradox? This analogy was introduced by Boris Beizer in 1983. He gave the example of treating fields against pests. That is, if you treat fields with a certain pesticide, not all insects may die. Conventionally, 80% of pests will die, but 20% will stay. And if you re-treat these 20% of insects with the same pesticide, they are likely to be immune to this poison.
It’s the same with testing. Repeated use of the same test cases that are not regularly reviewed or updated simply won’t be effective for existing bugs. After all, within the scenario described in the test case, you have most likely already found all the bugs.
But if you take a small step sideways, you may find that there are other bugs lurking there as well. This is why it is so important to regularly update your test documentation. But since we don’t have it at hh, and everything is tied to exploratory testing, the pesticide paradox in our work is quite rare.
And due to the fact that we do not write and regularly update test documentation, our testing speed is increasing slightly, but still.
We use Jira to introduce new bugs. In the bug ticket, we enter everything necessary for the bug to be easily reproduced and localized as much as possible. This can be logs, screenshots, videos, test environment, etc. Of course, a precondition, steps of replaying, expected and actual results must be present there.
Our testing tools
One of our main tools is sniffer traffic. They can be different. They can be Charles, Fiddler, Mitmproxy, and so on. We use Charles, simply because we are more comfortable with it. It’s not better or worse than other tools.
First of all, we use it to view the network requests from the application and the responses to them. And also to modify those requests and responses. Sometimes instead of making some unusual request from the application, it’s easier to replace it manually. Besides, we use different features like Throttling or proxying to testbenches. In general, anything that can help us in testing.
The next important tool is Postman. It’s a bit similar to the previous tool, and we use it to send network requests to the API and get responses. It’s not always the case when the interface is implemented, but we want to see some request right away and see what the API will return for it. That’s exactly what we use Postman for. In addition, we use Postman to send pushes to our test builds if we need to check some particularly custom one.
We also use test benches. This is a complete copy of the production, but with test data. The test bench has all the services that are currently in the production, they are always up to date, if you update the bench regularly. The tester can switch different services on or off, generate any test data, or even break the bench completely. But that’s okay, in that case they will just create it anew.
In addition, the bench has test data generation via fixtures. Basically, it’s a wrapper over the API that allows us to create any test data we need. It’s much easier than making them through the interface.
It is also worth mentioning the debug panel. The debug panel is a part of the application which is only available in test builds. With it testers can enable certain features, generate test data, see logs and many other useful things. If you are in a large company, and you do not have a debug panel yet, it is likely that after a while testers will come to you with a similar request. Because the debug panel really speeds up testing. And if testers don’t come to you with such requests, most likely, they don’t even know that it is possible. So it’s worthwhile to show them the value of this tool.
Other useful tools are adb and Android studio or Xcode for iOS development. I think these tools don’t need much of an introduction, so let’s move on.
A fact that worries a lot of people is that we have no detailed documentation on the product in hh. The feature description is just a layout and the product manager’s text about the main idea of this feature: what the analytics should be and everything in that vein. We also don’t have detailed test plans, test cases or test scenarios. The only test documentation we have is a checklist. It lists all the features that we have in our application so that when we deal with regress, the tester doesn’t forget what we have and check everything.
Yes, this sometimes causes some difficulties. We may change testers, someone may forget something. Sometimes there are questions — why does it work this way and not that way? But there are advantages here as well. If such questions arise, it means there’s something wrong here, and it’s worth discussing. After all, if everything works well, these kinds of questions are unlikely to arise.
Here our powers are over
QA in hh has the same powers as in all large companies with an advanced engineering culture. QA can block the release if they understand that there are some critical bugs, massive user complaints or crashes in the product. Also testers determine the severity of a particular bug. There are many factors for this: the functionality criticality, reputational risks, or its massiveness.
Besides, only QA has the right to merge a new feature into the develop branch. Of course, teamleaders also have this right, but they try not to use it just for no reason. This is done to ensure that nothing gets past the tester. Anything that has gone into the develop branch is guaranteed to be tested and reviewed by the tester.
QA is also involved in the process of releasing mobile apps into the stores. You might get the impression that testers are some reluctant egocentrists who think they’re the only ones who know the right thing to do. In fact, that’s not the case. Dialogue is very important in our company. You can always discuss the criticality of a bug with someone, be it a product manager or a developer. Perhaps some bug we can just accept, and accept the risks associated with it, and release it. But then we will definitely fix it.
A key quality of QA in our company is the ability to keep a balance between speed and quality. We try to work on a fine line when we can’t guarantee that there are no bugs in our prototype at all, that everything is perfect and nothing crashes. But we try to maintain a good speed of delivering the feature to the user with optimal quality. We teach this approach to all the new testers who come to work for us so that the tester is not an unnecessary headache in the entire development. However, we still try to maintain high quality standards in our application.
Release
The last stage in development is the release itself. As I said before, our testers are in charge of the releases. They create release branches themselves, make builds and put them in the store. Recently we’ve automated all this routine. How we did it, I told in one of our previous articles.
All of our regress is kept to a minimum: we have very good UI-test coverage. And, accordingly, before releases, testers check only the functionality that is not covered by tests, or which is better to check once again. Before a new app is released, we put it into the beta channel on Google Play. The beta usually has a limited number of users who are warned that the functionality may be unstable and with some number of issues.
With a small number of users, we can make sure that there is nothing wrong with our application. It’s not always possible to find all crashes and bugs at the testing or development stage, and here you can find a hidden crash. If there is one and it’s massive enough, we fix it and only then take it to production.
Fixing such a crash is a rather nontrivial task. And it is not always easy to reproduce it. Very often developers just look at stacktrace, see what the problem might be, fix it, and the tester has no way to check it all manually. That’s why we just re-enter the beta channel and see what happens. If everything is fine in beta, then the next step is to get it to production. In production we never go 100% at once either. First we put out the build at 20%, then we go for 50%, 70%, and finally we get to 100%.
We never put 100% in a rollout for all users at once. Google Play has a peculiarity: if you roll out an application to 100%, then the build that crashed cannot be removed from there anymore. In other words, this APK will remain there forever until it is replaced by a fresh version of the application with the fixed bug. If you roll out the build only 99%, then in the case of a massive problem, such a build can be removed from Google Play. The remaining single percentage is usually not affected. We have never had any complaints that users didn’t get the update.
The life after the release
After release, we make sure to monitor what’s going on with the application. First of all, we look at crashes. We have two important indicators: crash-free according to users and according to sessions.
We use tools like Firebase Crashlytics or Appmetrica to do this. We try to adhere to the figure of 99.9% of users without crashes. We just decided to set a value for ourselves and keep to it. It is counted for a certain period for a certain version.
In addition to crashes, we make sure to monitor feedback about the new version. And we definitely interact with our technical support team. We may receive complaints from users about some bugs, the tech support team handles them and, if the bug repeats, sends it to the development team. We reproduce it, look into the problem and fix it.
Also, all of our functionality is run under A/B experiments. Thanks to this, we can always just turn off something that isn’t working. This is, in essence, a feature-toggle. If there’s an experiment, we turn it off, the developers fix it, and we release the whole thing in the next version. And only then we turn the experiment back on.
Summary
Let’s summarize briefly. We build quality at every stage of a feature’s development, be it design-demo, development, testing, or release. The earlier the problem is found, the easier and cheaper it is to be fixed.
To make it easier for us to ensure the quality of development, we actively use autotests. We have them completely native for both iOS and Android. For writing iOS autotests we use XCUITest, for Android we use Kaspresso. We use our self-written DSL to generate the test data on Android. You can read more about it in our article on Habr (rus).
A little bit of numbers about our autotests. In Android, we have about 400 UI tests. There are 500 of them in iOS. Basically all of our UI-tests are written by testers, but developers are not forbidden to write them, either. For example, if UI tests are broken during the development of a new functionality or the improvement of an old one, they are fixed by the developer.
We have an inverted testing pyramid. Our main focus is not on Unit tests, but on UI tests. Yes, UI tests are more unstable than Unit tests, they take longer, but at the same time we trust them more because they repeat all user scenarios in more detail.
In addition to UI tests, we have Unit tests. Android has about 1,800 of them, and iOS has almost 2,000. Most of the tests are run at night. All of the tests are run on the branches of the features being developed, as well as in develop. In the morning, developers can see what the state of their developed functionality is. We mainly focus on develop. If a test crashed there overnight, that’s the first place the developer goes to look what happened. Perhaps there is a bug there or an unstable test. In any case, this must be fixed.
We always strive to make sure that the tests in our develop are green. This is very important for the test’s credibility. If they constantly randomly crash just because they are unstable, then at some point developers just lose trust in such tests. That’s why we routinely analyze crashes, fix our unstable tests, and develop our test infrastructure.
That is all. Feel free to write about your testing process in the comments and ask any questions.
Stay tuned.
