A journey to better test automation

Doing test automation right isn’t easy. And honestly, when you try to become good at it, it doesn’t make things easier in the first place. But similar to most investments for the future, it might hurt at first, but it pays off and gets you further in the long run.

At mySugr, test automation is an integral part of our work, as it is for any professional app or software company. But on top of that, parts of our app are classified as a medical device, which means that we are under regulatory control by authorities, therefore we care about quality even more. Facing these challenges we had to develop strategies within our agile environment to keep QA work doable. Naturally, things get more complex as your app grows.

Important questions were raised:

• In general: why doing test automation?
• Can we get along with just unit tests? Is UI testing doing the trick? Or how do we get a proper distribution of the tests?
• How do we manage a huge automated test suite?
• How can we create and maintain tests more efficiently?

Spoiler alert: we apply principles of clean code on testing. And for that we also have the new “sweetest” testing framework in place which is written in Kotlin and has just been open-sourced, and there is an intro on this blog, too; feel free to have a look!

But why even bother?

Some parts of our app needed to be done quickly so we needed to account for compromises. As a start-up, being quick can decide between life and death. So for these parts we sometimes had bad test coverage, which gave us some headache afterwards when we had to maintain and change stuff in the app. Refactoring without tests is like surgery with bad anaesthetization: it might hurt a lot and usually leads to complications. It affects you whether you’re a hobbyist coder or a multinational, but the bigger and older the app, the more it will hurt.

We relied on manual testing too much, which made us quite slow. No matter if you just run the app on your own to test a new feature or you have a department at hand which does it in a more professional manner, you always can fall into that trap. When you have to run your app to see if something is working as intended you’ve most likely already have fallen into it.

So what will this be about?

In this article I cover:

• Advantages of automated over manual testing
• How a good test automation strategy guides code design
• How we use different levels of testing
• What we believe is a good test and test architecture
• A new open-source testing framework that puts it all together

Manual vs. automated testing

Automated testing is superior to manual testing in many ways. There is still (and there will always be) a need for manual testing, especially exploratory testing. But manual testing doesn’t scale well and QA people usually can’t tell you what and where exactly something went wrong in the code. Furthermore, continuous integration can speed up your development vastly, and manual testing can’t be integrated in that process. It’s also difficult to test data-intensive applications manually, or to test features that need a simulation of a scenario over multiple days or months.

Tests guide your development

It’s a big mistake to see test automation in software projects just as a QA or testing tool. Unit, integration and UI tests are in fact great design tools, and they also help to focus your efforts and increase maintainability. That way it speeds up development not just in the long run but from the initial stages. It’s not easy to see the benefits in the first instance, but I strongly advise to have a look at test-driven development and try to understand what it is really about. Most developers agree it’s a meaningful practice, but only a few give it a real chance. I like to see the test-first practice as a necessary progression step in a developer’s career and thus substantial for any software company.

UI testing?

In the beginning we thought that automated UI testing might solve our problem. It turned out that testing a mobile app end-to-end is not straightforward… Well, everyone makes mistakes! It costs a lot of time to write these tests, they tend to be flaky and when something breaks it’s very hard to analyse where the problem lies. That’s especially true for mobile apps in comparison to web apps.

But we learnt from our mistakes, so there is now a different approach: we test the UI of our app separately from the underlying business and presentation logic. Therefore, the sole purpose of a UI test is to check if the UI interacts correctly with the underlying presentation logic. There should be no business logic involved. I’ll cover this topic and the reasons in more depth in a separate post in this series — stay tuned!

So, no, UI testing is not the ultimate answer in mobile test automation!

Distribute tests better

A much more important key for us is to have a proper distribution of test levels, which is the best place to start. It makes no sense to start with UI testing when there’s still a lot to do at the unit and integration test level. The testing pyramid, as described in an article by Martin Fowler, is our tool of choice to visualise the thought:

Source: Martin Fowler

• The basic idea is: unit tests are the cheapest to write, so we try to do most of the test automation at that level, may it be in the presentation, business logic, or data access layer of your app.
• Code integration tests are tests that check for the interaction between the units. They are more complicated to write but don’t have to be exhaustive, so we try to push edge cases down to the unit level wherever possible and just go for some easy cases on the integration level (e.g., a success and a fail case per functionality).
• System integration tests are similar but they run against the system, so the question such a tests should answer is: does the OS and the app communicate correctly with each other? These tests might be even a little more complicated to write and are definitely much slower as they have to run on physical or emulated devices.
• UI tests are an extension of this idea, they run on the operating system too but we are communicating with the user interface instead of the system API. UI tests are the most expensive to write, analyse and maintain and you should really have the proper strategy in place in order to make effective use of them.

There are tons of articles on how to write proper tests in a relatively time-saving manner. Feel free to have a look and explore these concepts!

What makes a good test?

When you have placed your tests all at the right level there’s still the question how you notice a good test. Let yourself be guided by these questions:

• Is the test documenting the behaviour of the system under test?
• Is it easy to read and self-explaining?
• Is it well-factored like all other production code?
• Are code duplications kept to a minimum?

Of course, there are lists of other obvious factors like speed and predictability which are very important (you can find out a lot about that on the internet). But I’m trying to make a point here: We found out that, after taking all the obvious rules into consideration, we still couldn’t answer all questions with “yes”. So we came up with our new testing framework “sweetest” which facilitates exactly these questions.

“sweetest” framework: organise tests better

The aim of this framework is to reuse test code and make it easier to create test setups. It’s written in Kotlin and makes use of the nice language features it’s offering, where the possibility to write DSL-like code is especially worth mentioning. Reusing code comes with the cost of extra structures, of course, but we also end up with maintainable code that is easy to read and extend. With that in place, adding an integration test on top of a unit test or adding more edge cases are often just a matter of a few minutes.

This is an example on how a test with our neat little framework looks like:

AuthManager needs a bunch of other objects to function, namely a SessionStore and a BackendGateway. The sweetest framework automatically generates a mock by means of the mockito framework for those. That’s of course a hint to dependency injection, which is naturally supported and it works perfectly together with DI frameworks like Dagger. In fact, it does its own dependency management, so you just have to tell it if a specific dependency should be initialized with a mock or a real instance. So, with an integration test, for example, you would just add additional dependencies to be real instead of mocked and then go on with putting together the already existing building blocks.

In order for all that to work, of course, you have to write these building blocks and put them into a class. This class then contains all mocking, setup, interaction and assertion logic for one class under test. I have to admit that it’s some extra work up front, but as I said earlier, that can be true for investments into the future; and with some template magic and the proper use of handy built-in IntelliJ features we can keep it to an absolute minimum. As always, it’s a matter of whether you see the benefits for yourself or are able to communicate them to your team.

In the follow-up post in this blog I’ll show you that you just need two lines of code to configure an integration test setup with sweetest, so stay tuned and let’s bring good architecture to test code!

What’s next?

If you want to know more about our test design considerations and see sweetest in action, there will be more on this blog, like on

• how to use IntelliJ to make writing test code fast and easy (download the live template here)
• how to write good test code efficiently (design considerations)
• how to integrate Cucumber
• how we do UI testing
• how the framework works on the inside

Also feel free to have a look at sweetest’s source code, try it on your own, and of course to fork and contribute — it’s highly appreciated!