Unit Tests

Reducing flakiness in Swift tests

• Flakiness is what happens when tests don’t run consistently — when you get different outcomes depending on time, which machine they’re being run on, or whether they’re run on CI or on the computer you work on.
• Writing test is about making assumptions, but some times we go too far in making assumptions.

Things that cause flakiness and ways to fix it

• Having wait times. Dispatch.main.asyncAfter(deadline: .now() + 5) {}. You should remove the wait times and one way to do that is by creating a protocol (you can see an example of this in the article).
• Jumping between queues, such as background thread and main thread for UI updates. The solution given was just to perform everything on the main thread. The solution give is below.

func performUIUpdates(completion: @escaping () -> Void) {
    if Thread.isMainThread {
        completion()
    } else {
        DispatchQueue.main.async(execute: completion)
    }
}

• Waiting for elements to appear on screen for UI tests. Xcode UI test runner takes into account animations and has an internal wait time (you’ll see Waiting for app to idle in the logs), but custom animations will through this off. So the solution is to wait for an element to appear with a small extension to XCTestCase func wait(forElement: element: XCUIElement, timeout: TimeInterval). Even though we just talked about timeouts above, _____ says the exact timeout is not really important since UI tests can sometimes be really slow to execute. So it doesn’t hurt to add a couple of extra seconds to avoid flakiness.

Test Smell: Everything is mocked

Pretty good short article, just wish it had more relevant iOS terminology and examples.

• when to not mock your class: it has only accessors or simple methods that act on values it holds, it doesn’t have any interesting behavior.
• Don’t mock third party libraries directly. Define an interface for the service that directly applies to your needs (integration tests).

5 Questions Every Unit Test Must Answer

• Before you implement, write test first. TDD.
• Failing tests should read like a high-quality bug report. What is in a good test failure report? 1) What are you testing? 2) what should it do? 3) What was the output (actual behavior)? 4) What was the expected output (expected behavior)? Which I think is something we struggle with at Hudl. It’s usually clear what test broke, but deciphering what part _____

What are you testing?

The compose() function takes any number of stamps and produces a new stamp

What should the feature do?

Write a simple string in the test as a clear focus on the requirement(s). 'compose() should return a function'. What component aspect are we testing, this will differ based on the granularity on the coverage of the test. We are to test the return type of compose(), to make sure it returns a function instead of undefined.

test('Compose function output type.', assert => {
    'compose() should return a function.'
});

If the only available assertion in every test suite was `equal()`, almost every test suite in the world would be better for it.

The writer states this is because it answers the two most important questions about the unit test; what is the actual output, and what is the expected output? If you finish a test without answering those two questions, you don’t have a real unit test.

test('Compose function output type.', assert => {
    const actual = typeof compose();
    const expected = 'function';
    assert.equal(actual, expected,
        'compose() should return a function.');
    assert.end()
});

I thought the above article was pretty interesting but was geared towards javascript, so I decided to do a more iOS relevant example. In doing so, I learned that you could also write and run unit tests in Playgrounds.

Writing unit tests in Swift playgrounds

So here’s an example of writing some simple unit tests for NSNumber and NumberFormatter

Four simple rules for architecting iOS unit tests

Isolate components

• Tests one behavior of one component at a time.
• You should have one XCTestCase file for every component
• Have a good naming convention for your tests and files. class PayrateValidator is in a file called PayrateValidator.swift, which has a test class PayrateValidatorTests which lives in a file called PayrateValidatorTests.swift.
• The key to writing good unit tests is understanding that you need to break your app into smaller pieces that you can test.

Isolate Behaviors

• A well-architected unit test suite breaks each component into behaviors, each with its own function to test them.
• each function should only have one XCTAssert()

Do include inverse behavior

• You want to know that your component does what it should do and also doesn’t do what it shouldn’t do.

Do be descriptive

• methods should be descriptive. Exactly what is being tested and how the component is expected to behave.
• A well-architected set of unit tests should function as documentation.