How many tests should I use?

I've started this blog with an explanation of how to improve the quality within the app. On my way there we've encountered a lot of different types of testing. But how much of each type should we invest into our app? As always there is a balance to strike. This time it's about how much we can invest vs how much value it will create (isn't it always?). So let's have a detailed look.

Unit-Tests

We've looked quite a lot into unit tests. How to do it in Objective-C, in Swift and how we can test Apple's MVC (don't forget calling controller.view on a UIViewController when testing UIElements). What are the advantages of Unit-Tests?

• fast
• reliable (if you do it right ;))
• cover a lot of parts

With all the advantages, what's there to miss?

• they are isolated
• don't test whether parts fit together

The most important part is them being fast. It doesn't matter if we have 10 or 1000. They run within less than a few seconds. Developers won't have a reason not to run them before committing changes.

It's quite simple. The more we have, the better. This is also a reason, why we use them in TDD. Make sure to write them independent of everything. Don't use databases, or networking. Instead stub it!

UI-Tests

UI-Tests have also been a topic. We've seen multiple frameworks to implement them and also how to make them independent of any backends.

The advantages:

• Test components
• You can use them for acceptance tests
• They behave similar to users

As always you can expect them to have disadvantages:

• slow
• fragile (every change to the app can break them)
• slow (did I mention it already?)

UI-Tests need a lot of maintenance. They can break any time, even just due to the simulator not being able to connect to Xcode. So every failure they show, might not be a bug within your app. You'll need to investigate it. And most of all, they run very slow. Even when telling them not to restart the app at the beginning of a test, using a mock backend and removing animations (which you still shouldn't do) they take forever. Currently our 50 tests need more than 15 minutes to run. No developer will ever do it after every change. So your CI needs to.

It's nice to have them. But be aware they require a lot of work. Since they are slow, there shouldn't be as many tests as Unit-Tests.

Acceptance-Tests

We've looked into UI-less Acceptance-Tests. It's incredible what you can achieve with this type. They can even help you working with your client and nail down requirements in a more precise way (look into BDD if you want to know a way to work with your client).

So why we should use them:

• Fast (compared to UI-Tests)
• Test integration
• Everyone can write tests (not only the developers)
• Everyone can access the result (without knowing the CI)

But as always there are things preventing us:

• It requires it's own UI-code (called fixture)
• Everyone on the team has to agree on using them
• depending on the infrastructure it gets a little complicated

They are blazing fast compared to UI-Tests but compared to Unit-Tests the speed is less so. The setup is not as easy and it will take time to convince your team and client to use them. But they are totally worth it, as they test the interaction between components.

Golden-Master-Tests

Golden-Master-Tests one is kind of in between Unit-Tests and UI-Tests. In the end they help you identify changes in the behavior of your app. This doesn't mean these changes were unexpected, but they will be listed.

Advantages:

• Checks your UI even for smallest changes which you can't see
• Easy to integrate into any kind of project

Disadvantages:

• You don't know what is wrong if something breaks
• Will break with every change to the UI

Our app should have some UI, but as you know, less is more. If the user is able to do a job without extensive navigating, it's better. So our UI is limited and thus it should be a small number of these tests. They are quite fast and if you don't have any other option to verify your code, use them extensively. But if you already use Unit-Tests and other types, you can shrink the number to a bare minimum.

Chaos-Test

Netflix really did a fast one on us with Chaos-Testing. This technique allows us to check for a lot of possibilities. Sadly we are limited in our debugging options.

Advantages:

• Tests nearly every possible interaction
• Tests nearly every possible value

Disadvantages:

• In case we test via the UI, we really don't know, what happened
• Time intensive

While testing properties is rather fast and can be reproduced, we will have a problem, in case we run a test monkey. It does enable us to find crashes, which the user will otherwise produce, so it is better than expected. At the same time, we will have to invest a lot to reproduce the bug. In case you use it, run it nightly, this way you test extensively and it doesn't show up during a release.

Manual-Tests

Everyone of us did them, and every one of us hates them. Still they are useful as we've learned in "Manual Testing is Dead".

There are still some advantages:

• Using tours you can test things, either not related to developers or specific behavior of the app
• Some things just are a pain to automate (testing whether the interaction with a casheer system works)

Disadvantages:

• Time
• no one likes them, so they are done without care
• Why repeat them, if we've automated the test cases?

As always the most important factor is time. If a test needs a long time by hand, it's often skipped or not reproduced. Testers will act sluggish whenever they work on a testplan, if it's too long. Spending half an hour every release is no problem. Spending 4 weeks is!

So keep your manual testing effort low.

Conclusion

So far the recap of the different types. Everyone probably heard of the testing pyramid. The volume of every step describes how many tests there should be of this kind. Ours looks like this:

As you've probably expected Unit-Tests are the master of all tests. Afterwards there are Property-Based-Tests, Acceptance-Tests, Golden-Master-Tests, UI-Tests, Chaos-Monkey-Tests and Manual-Tests. As always this is an opinion for a green field project. You will have to adapt these, whenever you work on a different project, but in an ideal world, strive towards this pyramid.