How to leverage Acceptance Tests in your iOS Apps

Mobile development is quite a new discipline compared to others like backend development. Since Apple didn't care about testing for a long time, the community started late with testing. In the beginning we had Unit-Testing tools, which I've looked into at Unit-Tests in Swift and Unit-Tests in Objective-C. Having unit tests is great, but as the name says, it's just testing units. So we kept on looking and found Golden Master Testing. Another great tool to improve our quality. Finally we could ensure our UI's to be correct. But does having a correct UI mean our app is doing what it's supposed to do? So how are we supposed to test our entire app? We've moved on to UI-Testing. Great finally we can ensure our app to be correct. Having written a bunch of tests for our app, maybe 2 years in (including 2 iOS and Xcode updates) we realize:

UI Tests are slow, flaky and hard to maintain

Sooooo is this the end? Do we have to live with these slow tests, or can we somehow improve them? Let's have a look why UI-Tests behave like this.

UI Tests

Speed

Apples solution in UI-Tests was to have two separate processes running.

Whenever tests requested an XCUIElement, the TestProcess fetched the entire application state from the original app. Then it would execute the search query through the entire state and list all matching elements (often only one, but still would continue after finding the first). Knowing all these steps we can say, where within this architecture time was needed:

• Creating AppState as a tree for XCTest
• Transmitting AppState to TestProcess
• Querying the entire AppState

With Xcode 9 things changed a bit. Instead of executing the query within the TestProcess, it transmits the query and the app is executing it. Furthermore you can use ".first" to find the first element within the tree and then return immediately.

This makes UI-Tests faster, but it is not as much of an increase of speed we hoped for. Instead of running for 60 minutes, they might need 52. Another reason for slow tests are animations. You can't check whether they are true, so there is an option to deactivate them:

UIView.setAnimationsEnabled(_ enabled: Bool)

I don't like this approach, as you need to implement it within your production code and there might be bugs only showing up when using animations. In the end you will have to consider the benefits and decide if you want to risk it or not.

Stability

Having looked into speed issues, why are UI-Tests flaky?

Since tests run in their own process, we know they have to synchronize. But for this the processes themselves need to find each other. Whenever this is not the case, the tests just fail for no reason within your app. This can not only happen during the first start of the tests, but also in the middle. Whenever your app restarts, the app process needs to be found. Furthermore sometimes due to unexpected app behavior the tests run out of sync. This might just be a system dialog popping up und you not handling it, or a small error within your network connection, but it can be devastating for the end result. Knowing our tests not being stable means endless hours of debugging whenever a run fails. It might be the test, but often it's just some kind of timing, (within the test) unexpected behavior, or just Xcode acting up. Still this is time consuming.

Maintainability

In Screen Objects we dove into how to maintain tests. Even though we try to adhere to them, I often see code, which is not independent of the app itself. Function calls like .tap() prevents the tests from being loose. A small change in the UI might just kill all the tests on this screen. Mind you, it's often easy to fix. Still it takes a lot of time.

I know I'm criticizing developers here, but we all know, no one is without fail. In the end we will often stumble into tests, which are tightly coupled.

Acceptance Testing

Now that we've looked into a lot of reasons against UI-Tests, what can we do to improve them? For this we need to ask ourself, what we actually want to achieve. In my opinion the most important question to be answered is this:

Are we writing the right code?

Unit-Tests will answer the question, whether our code works in isolation, but what prevents us from writing code resulting in this:

Our code works, as do the windows, but when putting it all together, suddenly they don't work. This is what UI-Tests test for us. They check whether our code not only works, but if it is the right code to work in unison with the rest of the app.

We've identified two main reasons for our issues with UI-Tests. One is the separation of processes and thus resulting in a state which is out of sync, and the other is for the UI to take forever, when we try to achieve something. Both can be traced back to the UI.

One solution would be to write a special UI for our tests, which has no graphics and can be reduced to basic values.

As an example project we will use Connect Four. Instead of telling our UI-Tests which position (or button) to click, we could just have a command line interface and input the line, which will return the result.

Let's have a look how this could look like:

Player 1:
0
Player2 :
1
Player 1:
0
Player 2: 
3
Player 1:
0
Player 2:
1
Player 1:
0
Player 1 wins

With this example we could create different tests:

• Input a whole sequence (0,1,0,3,0,1,0) and expect a result (Player 1 wins)
• Input just a few values (0) and expect a result (Player 2:)
• Input a draw

And many more. Testing in this fashion would speed up our tests a lot, but at the same time we would lose out on testing the UI (which isn't necessarily bad, as we also have Golden Master Testing and still could write some UI-Tests).

Fitnesse

As mentioned above, testing doesn't have a long history within development for Apple platforms. Luckily there are languages who have the necessary tools. In Java we can find Fitnesse. Originally it was a Java only tool, but luckily, good tools tend to be ported to other languages like C. Having Objective-C as a strict superset of C is quite useful in such occasions. And since Swift is designed with the idea to incorporate all the existing Objective-C API's it's still able to include C libraries.

Architecture

Fitnesse consists of a Wiki containing and running the tests, and a small server communicating between the app and the Wiki.

As we designed above we can insert some kind of input, and receive a result. For this to work we have to write a corresponding UI (called fixture), which replaces the UI during our tests.

Test Decision Table

Let's design our tests for Connect Four real quick. We will use a decision table containing a column with our input, and a column with our output:

|Turns          |Result         |
|0,1,0,2,0,3,0  |"Player 1 wins"|
|0              |"Player 2:"    |
|               |"Player 1:"    |
|0,1            |"Player 1:"    |
|0,1,2,1,2,1,2,1|"Player 2 wins"|

Turns represents our input separated by commas and result is the expected app state. We will skip the full board draw possibility or other options, but you can add them, if you like!

Wiki

Before we write our first fixture within the app, let's have a look at the Fitnesse Wiki.

We can easily add a new test listing by adding a new TestPage. Within this page our tests are being listed. Luckily the format for our tests, is the same as the above used decision table. So we can just copy and past it :) The top line represents the fixture the tests needs to call.

Clicking on Run, will execute the tests and return the results within seconds, not minutes.

Fitnesse Integration into an existing project

Slim is the RPC-Server used to integrate Fitnesse into your app. The Objective-C port is OCSlim. We could do it the manual (very complicated way), or simply use CocoaPods (and have a few more steps). Using CocoaPods we have to do the following:

• Install Xcode Templates
• Add Acceptance Tests Target
• Execute CocoaPods

To install the Xcode Templates is rather simple. Download Download OCSlimProjectXcodeTemplates from GitHub and run make. You should see two new types of testing templates within Xcode:

• Acceptance Tests
• Acceptance Unit Tests Bundle

The difference is Acceptance Tests are being run by the Wiki and the Acceptance Unit Tests Bundle will report the result within Xcode.

The next step is to add the 'AcceptanceTests' Target. Make sure to call it this. It should work with a different name, but I didn't get it to run at all otherwise.

Now add the following to your Podfile:

target 'AcceptanceTests' do
     platform :ios, 9.0
     pod 'OCSlimProject'
 end

When you open the workspace, it might complain about having the wrong Swift version, but this can be fixed by the auto converter.

Running the tests needs you to run the Acceptance Tests target, before you can run the Wiki tests. Furthermore node and ios-sim need to be installed:

brew install nodejs
brew install ios-sim

Writing Fixtures

When running the tests within Fitnesse, we get an error:

Could not find class ConnectFourAT.

This can be easily fixed by adding a class to our Acceptance Tests Target.

@objc(ConnectFourAT)
class ConnectFourAT: NSObject {
}

In case we don't have a decision table yet, our tests are green and we have a working connection between the Wiki and our app.

Next let's add the above decision table and run the tests again. We get failing tests:

Method setTurns not found

To fix this, we add our turns variable and a function which returns our result

@objc(ConnectFourAT)
class ConnectFourAT: NSObject {
    var turns: [Int]
    
    func result() -> String {
        return "Player 1:"
    }
}

The Result in our Wiki Tests is our expected outcome. For this to be obvious tobFitnesse, we have to use a ‘?’. So rename it to ‘Result?’.

At least one of our tests will now pass.

Starting with this, you can continue adding the interface on your fixture and evaluate values, when they are set.

Other Tools

Of course there are other tools than Fitnesse. Some of them are Gauge and Robot Framework. Have a look which ones you prefer. Since there are a ton of them, make sure they don't use the UI for testing.

Conclusion

We looked into different reasons why UI-Tests aren't the perfect solution and how to improve the situation. Acceptance Tests provide a good solution for most of the issues we have, but don't replace them entirely. For those of you, who might be wondering, why you should introduce such a setup, if you could only use it with iOS, you might be happy to hear you can also use it with Android.

Furthermore on the same instance, you could run also the Fitnesse tests for your Java backend.

There is one issue though. When running it with Xcode 9, the simulator gets restarted on every run. Prior versions didn't have this problem, and I guess it will be fixed soon.