Top 5 Technical Director’s Questions About Automated Testing

Hey everyone! I’m Olya, mobile app tester at hh.ru. We continue the series of the top questions about automated testing. We answered questions of manual testers and managers earlier. Now it’s time to answer the top-5 of the scariest questions from technical directors: how to choose an automatization framework and win, hiring difficulties, and labor costs for autotest support.

The video version of my answers is available here.

Question #1. How to choose an autotest framework?

Let’s start with a story about how we selected a framework for android autotests.

Once upon a time, a team of hh.ru’s developers and testers decided that it would be nice to automate regress. Here comes a large and detailed research.

We were learning what existed in the market:

• what issue those frameworks users face,
• how quick and how well those issues are solved,
• how often do you have to use kludges for autotests to work in a stable manner,
• relevance in the community

In the end, we’ve got a pros-and-cos table for each framework, which we were going to choose from.

We determined what was of greater importance to us. For instance, to write autotests easily in order to make them readable and easy to understand for testers and developers. So, we excluded unsupported, complex, cross-platformed, unstable frameworks and chose Kakao.

Kakao is a framework for the well-known Espresso. Writing autotests on it was much easier and more convenient. At least, element detection didn’t take us 3–5 lines of code, as it did in Espresso. We had been automating on Kakao for about half a year, and over that time we got a clear idea of what we were lacking in the chosen framework. So Kaspresso came along, which fulfilled all our desires.

Kaspresso is a more convenient framework. It helps not only to write autotests easily, but also makes them stable and readable. Under the hood, Kaspresso has a lot of useful functionality, thanks to which even the most complicated mobile app checks are written quite easily.

In addition to everything written about the benefits and delights of Kaspresso, we have added steps — a human-readable description of what we do in a test or in a method. From these steps, a step-by-step report is generated — you can see what steps the test passed and at what step it failed. We also made dsl to create test data. You can read more about it in this article.

It’s all much shorter and easier with iOS. Everything here is out of the box — XCUITest, with which everything works perfectly and stably. The framework is up-to-date, supported, stable, and convenient. We didn’t have to try any other possible options, because XCUITest meets all our requirements.

Question #2. How is life with automated regressions for you?

Over the two years that we’ve been living with automated regression, we have a solidly established weekly release train. Releases are regularly sent to production without manual regression thanks to quantity, quality, and, most importantly, trust in autotests.

In addition to launching autotests on release branches, the entire set of autotests is always run on the dev feature branches before the merge. No code that has dropped tests is allowed into the developer/master. Because of this, we keep the development stable, we have confidence in it, and we can cut off the release branch at any time.

For the tester, automated regress can mean more free time for more important and useful tasks — such as writing new autotests, increasing the stability of existing tests, developing their direction, and similar cool tasks.

Question #3. Does it take a lot of time to support autotests?

Support for autotests is built into our development processes. How it works: every week a duty tester is assigned, one of whose tasks is to monitor the stability of the tests. If a test is dropped, the person on duty must figure out the reason for the drop and assign someone responsible to fix it. If the test dropped because of its own instability, It’s the tester’s task, if the test found a bug, It’s on the developer whose code has affected the autotest. According to experience, most often it does not take much time — one tester spends a few hours in a week.

More complex tasks are development, improvement and optimization, and infrastructure fixes. Such tasks go along with the development of product functionality and have a corresponding flow (elaboration, decomposition, assessment, development, etc.). This does not happen too often, on average once a quarter per platform. Note that most often these are not high-priority tasks, and they are done when the team has the resources to do them.

If you set everything up clearly, learn how to work with flaky tests, and avoid them as much as possible, then supporting autotests takes little time and brings plenty of benefits.

Question # 4. How hard is it to find a mobile automator?

Sometimes it’s hard to find just a good mobile tester with relevant experience, and things get even harder if automation experience in a particular technology stack is required.

In hh.ru mobile teams, we usually look for just cool manual testers who want to develop towards automation. During interviews, a basic knowledge of programming is certainly a plus, but not the decisive factor. We are willing to teach automation in-house. But what really matters is that a person must be eager to develop, able and willing to learn, and be proactive in these aspects. Of course, it’s difficult to accurately determine this during interviews, but it’s not completely impossible either.

Question #5. How long does take for a manual tester to become an engineer of automated testing?

Again, based on our experience, we hire a person with no automation experience and for the trial period (3 months) he is tasked with writing his first autotest on whichever platform he likes the most or seems easier. And no one has ever failed a probationary period.

Naturally, the fact that a person is not writing autotests completely from scratch makes a big difference. We already have ready-made autotests that you can look at and write by analogy, and people who are willing to help and answer questions.

As a result, in 3 months, we get a person who already understands how to write autotests for at least one of the platforms. The next step is to write the same test for the second platform. In another 3–4 months we will get an independent automator of mobile applications for both platforms, who may need help with some complicated things for a while. But he will be free to write simple autotests for both platforms in six months.

Instead of a summary:

This article is a follow-up to the series of answers to questions about automation. Next, we’ll find the final TOP-5 questions of a junior automator about autotests. Let’s talk about flakiness, stability, and bugs from production.

If you don’t want to miss new videos, articles, and other news, subscribe to our news channel on telegram and “HHella cool stories” channel. And you can ask our engineers any questions on any topic in the comments, in hh developers’ telegram-chat or in DM.

Stay tuned!