How to use Chaos in your iOS Tests

Have you ever heard of the infinite monkey theorem? It goes as follows:

“If you put a monkey at a typewriter hitting random keys, given an infinite amount of time it will bash out any expected written text”

There are also other versions which use an infinite amount of monkeys instead of time. As app developers, we have a similar situation. Our apps are used by a large number of users (okay they are numbered and not monkeys but still.. it’s similar) who are using our app in ways we can never imagine. As a result, they will find bugs we've never anticipated. So what can we do to prevent this?

Netflix coined a new term: chaos engineering. The idea is to introduce chaos into production code to find out how the system behaves while malfunctioning. They published their chaos monkey, which kills instances of their life system. We might be able to use this for our backend system, but in general, our apps have their own problems and we can’t just kill them while a user uses them ;).

So what can we do?

Unit Testing

When writing Unit Tests for algorithms we normally have to cut back on the input values as these can be rather large. Instead, we think about corner cases and tests these. Sadly this can result in us missing quite some bugs. Considering the idea above of adding controlled chaos into our tests we could start using random values within a specific range. These values need to be generated and our algorithm has to hold a condition (meaning the condition is always true).

Luckily we don’t have to write all these randomnesses ourself. SwiftCheck has been created to do this for us. Let’s dig in!

While reading the following explanation consider Email as our domain struct:

struct Email {
  var local: String
  var host: String
  var tld: String
}

Generators

In SwiftCheck you can create arbitrary data by using generators. When trying to generate email addresses we have to generate three parts: local, host and TLD. The local parts can consist of uppercase and lowercase letters, numbers and certain kind of characters.

These can be combined:

.one(of:) will pick one of the provided generators and return a value of this generator.

There is one more condition. The local part is not allowed to have a “.” at the end:

Next is the hostname:

For the TLD we reduce the number of possible TLDs to existing ones (and as these are still quite long, let’s limit them in this example to five):

Using these generators you can generate arbitrary email addresses. To do so you’ll have to conform to the Arbitrary protocol:

This is already quite a lot of code. But still, there is no testing in sight. That will be next on our list, but just a quick reminder. This is all testing code and should be within your test target. You don’t have to add the extension in your production code, except you want to use it within your app.

Properties

I promised you to use chaos within tests. So why am I suddenly talking about properties? Properties in SwiftCheck are not your usual properties within structs. Instead, they are properties of your app. In a way, they describe your app and must always hold true.
Let’s look at our above email. Imagine you have written an algorithm checking whether an entered email address is correct or not. We could just enter a lot of random email addresses into the algorithm and test for failure:

Assertions

What happens when an error occurs? We don't know the exact input data making the tests fail. There is no real option to know, what happened. This problem is also solved by SwiftCheck. The result will look e.g. like this:

*** Failed! Proposition: email addresses come with a TLD
Falsifiable (after 472 test):
ArbitraryEmail(getEmail: "4@meq")

The first line represents our property, the second how many tests were executed until the error occurred the third lists the input value. Sometimes it also lists the seed used to generate the value, so the tests are reproducible.

Let's imagine you have an array with multiple 1000 values in it as test data and the test fails. It wouldn't really help to have this array listed. So SwiftCheck has another method in its Arbitrary protocol called shrink().

This method provides an option to reduce the value. What SwiftCheck does in case of an error, it calls the shrink method, which reduces the array maybe by half, checks the test again, and repeats until it reaches a minimum.

Having this minimal dataset you might have to check for maybe 20 values instead of 1000, which will make everything quite a lot easier.

Other Tools

SwiftCheck is the Swift implementation of a Haskell tool called QuickCheck. There is also a version for Objective-C in case you can’t switch yet. It’s called Fox.

Next, to these QuickCheck based tools, others exist to generate test data. One such tool is Fakery. The scope of this tool is way smaller, as it just generates data which looks realistic. How you work with it, depends on you.

Another tool is Dixie. This is an Objective-C testing framework which alters the behavior of an object. This can be useful in a variety of ways e.g. hijacking the localization system and returning odd strings.

UI Testing

UI tests can also make use of Chaos. I've experienced it before when starting a new job. It started similar to all other jobs. Setting up your computer, getting to know the company, etc. Just on the second or third day, instead of doing all this, I started supporting a colleague with a bug described as:

App does not respond!

That’s not a lot to go on and we didn’t have any debugging information. There were no crashes or logs at all and our only solution was to kill the app and restart it. The only time we had a device with this error, we were only able to see how it behaved. Which wasn't very much, namely: very very slow animations.

With not a lot to go on, we tried to reproduce it. You probably can imagine how much time we spend without any success. A lot of theories existed (which were all proven wrong later on), but without being able to reproduce it, how should we disprove them?

At one point we decided to write a monkey. Just a tool clicking for a specific period of time at random positions on the screen. Running it overnight, we reproduced the issue for the first time. With this information, we refined the approach and were able to reproduce the issue within 5 minutes. Afterwards, we were able to fix the issue and since then we didn’t have any large problems anymore with the app.

As I described above we introduced chaos into our app by randomly clicking on the screen. This is a similar approach we’ve used for unit tests. It’s randomizing the input to the system. While unit tests use precise values, UI tests use gestures and positions. Simulating these we can test our apps.

Looking into XCUITest we can generate user input in random places by generating a random coordinate:

In this way we could implement all different input events:

• Swipe left
• Swipe right
• Scroll up
• Scroll down
• Long Tap
• Double Tap
• Shake Device
• Rotate device

We could write all these events (and more), then create a test which calls them randomly for a specific amount of time. Or we could use Zalando’s SwiftMonkey.

SwiftMonkey

SwiftMonkey consists of two parts. One generating and executing the random UI Tests gestures and one displaying these on screen. It's rather easy to set up, but if you use Carthage you'll have to do a few extra steps. Currently, it's not supporting Carthage out of the box, due to having no shared scheme.

First of all, add it to your Cartfile and fetch the sources:

echo 'github "zalando/SwiftMonkey"' >> Cartfile.private
carthage update

Having the sources you can open the project in Xcode and in Manage Scheme you can select Shared:

cd Carthage/Checkouts/SwiftMonkey/SwiftMonkey
open SwiftMonkey.xcodeproj

With this done, we can build the libraries and include them the usual way:

carthage build

Having all this done, we can add a corresponding test:

I would go the extra mile and create an extra target. This way it's easier to control when to run the monkey tests and when not. Usually, you don't want them on every run in your CI, but nightly might not be too bad.

MonkeyPaws

In case you want to visualize the monkey's input, you'll have to add some production code. Integrate MonkeyPaws the same way as described above and then add:

Having done all of this will display the random input SwiftMonkey generates:

Conclusion

Even the best developers are unable to predict all of the states their app can have. This results in not having tested everything. Since nobody is perfect and we can't test everything, our apps are prone to have bugs. Finding these bugs is difficult, but Chaos testing can help. In unit tests, we can test a wide range of properties when testing our algorithms. In UI tests we can use monkeys to do things with the app, we've never imagined.

It sounds like a lot of work, but since this might prevent major bugs in production I think it is worth it. As I've described above it can even be used to find the bug in production. In case you are interested, it was the UIWebView. Starting with iOS 9 it seems like every navigation creates a bunch of gesture recognizer. When transitioning to a different page, these are not deallocated, thus remaining. The moment you touch the screen, all of these fire and will overload the main queue thus making your app nonrespondent. In the end, the watchdog killed it.

Previous: Golden Master Testing