Nobody expects Completion Inquisition!

How to test non-called completion handler scenario

Testing completion-handler API happy path scenario is really simple with XCTestExpectation. Things get complicated if your test case has to verify whether given handler was not called. It is still straightforward if it’s called synchronously after some other event that you emulate. But how about a case where execution is deferred in time? In this article, I will demonstrate you a solution to achieve 100% sure that given completion handler will never be called — fast and reliable.

Problem statement

Let’s talk about unit tests of some asynchronous task that provides completion handler — let’s say downloading JSON from a server. It seems obvious that we have to verify if a completion is called in a happy-path scenario. XCTest has nice API for that: just create an expectation, fulfil it in a completion and wait for all fulfillments at the end of a test:

I am pretty sure you saw such kind of tests plenty of times.

However, I don’t see that much test scenarios where we check that something did not happen. One example regards initiating download task that is invalidated/cancelled before it actually finishes (succeeds or fails with an error). At glance, there is nothing wrong with simple XCTFail() in a completion that shouldn’t be called. In case something was wrong (presumably due to a bug), we will have a change to fail current test and see red test in Xcode or CI. But is it really enough? What would happen if our Downloader defines@escaped callback and calls its later e.g. in DispatchQueue.async block?

Deferred execution of unexpected handler (Completion Inquisition)

Just to remind you, unit test always uses main thread to operate and after it reaches the end of a function, it is immediately classified as failed (red) or passed (green). Therefore, deferred execution of XCTFail() does actually log some failure but Xcode is not able to determine which previous test it actually corresponds to. If you are lucky, you will see errors inXcode console, but your entire test will be still green:

Test Suite 'All tests' passed at 2017-09-05 18:44:31.858.
Executed 3 tests, with 0 failures (0 unexpected) in 0.209 (0.215) seconds
2017-09-05 18:44:31.865211+0200 NotCalledUnits[66089:3985807] *** Assertion failure in void _XCTFailureHandler(XCTestCase * _Nonnull, BOOL, const char * _Nonnull, NSUInteger, NSString * _Nonnull, NSString * _Nullable, ...)(), /Library/Caches/com.apple.xbs/Sources/XCTest_Sim/XCTest-13188/Sources/XCTestFramework/Core/XCTestAssertionsImpl.m:41
2017-09-05 18:44:31.866778+0200 NotCalledUnits[66089:3985807] *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Parameter "test" must not be nil.'
*** First throw call stack:
(
0   CoreFoundation                      0x000000011184626b __exceptionPreprocess + 171
1   libobjc.A.dylib                     0x000000010dfa4f41 objc_exception_throw + 48
2   CoreFoundation                      0x000000011184b402 +[NSException raise:format:arguments:] + 98
3   Foundation                          0x000000010da81bd3 -[NSAssertionHandler handleFailureInFunction:file:lineNumber:description:] + 165
...
...
...

Isn’t it misleading? Succeeded test with a failed assertion in a console…

Naive solution

Another approach is to wait a bit and assume that as long completion did not pop up within, let’s say 0.1s, is will never be executed. It has two obvious cons:

• You have to guess how long to wait. Should it be 0.1s or 0.01s? If you set it incorrectly, it can easily end up with invalid or unstable tests.
• It increases single test duration which by default should be as quick as possible by default.

Both drawbacks may lead to a dangerous trap:

When your tests take too long or are unstable, your teammates (or future you) start ignoring them and sooner or later just ditch them.

One possible solution

Let me ask you a question: when do you have 100% sure that some external part of your system will never call a completion? Answer: In a case when no one keeps holds a completion block anymore!

To verify that nothing deferred execution of a block, we have to ensure that all other instances lost reference to a completion. OK, Sounds easy!

But how to implement it? Of course we cannot define weak reference to a block: ‘weak’ may only be applied to class and class-bound protocol types. So, let’s define simple class ReferenceObserver that just notifies its deinit:

Then, let’s modify a bit our previous test: initalize observer , bind its deallocation handler to fulfilling XCTest expectation (line 4) and capture reference inside our unexpected completion handler (line 8):

At the moment we have exactly two references to our ReferenceObserver instance: reference and capturedReference (in a completion). Note: we had to add dummy assignment to an anonymous variable _, otherwise Swift would not capture it at all.

As you can expect, let’s get rid of one of helper reference (just callreference = nil) and voilà:

Only completion handler block keeps reference to ReferenceObserver and once completion handler is removed from a memory, deinitCompletion expectation automatically becomes fulfilled:

Deallocation of completion handler automatically calls deinitCompletion.fulfill().

Here is our final test:

We ended up with a solution that:

• does not unnecessary increase test duration because waitForExpectations quits immediately, after ReferenceObserver deinit.
• ensures that completion will never be called.

You may be wondering, what does 0.1s mean at the end of this test? It is just a maximum interval you let Downloader instance to clear its handler reference.

Alternative notation

If you are like me, and you don’t like optionals in your codebase even in your unit tests, you can leverage additional scope to get rid of ReferenceObserver? optional type. Did you know that you can create a do section without any catch, and it creates only an extra scope? If we init our ReferenceObserver in a dedicated scope, its reference will be deleted at the end of it and thus we can remove an optionality. Once //Arrange section ends, the only one reference to an observer will be attached to a downloader. Below notation is equivalent to a previous code:

The price that we have to pay here is readability. Some developer, non familiar with an empty do scope construction, may not understand what and why it actually occurs here. Therefore, despite my general reluctance to an optional, I would choose previous notation with ReferenceObserver?due to its readability and clearness.
What is your opinion on this? Do you agree with me? You can comment below or catch me on twitter, @norapsi .

Sidenote
At the end, let me here remind another golden rule about writing unit tests:

Every time you write a test, verify that if you modify implementation to simulate a potential bug, your tests will actually fail!

BTW. Such modified test is called a mutation test.

This is especially important for described above solutions. Please keep in mind that if you don’t correctly capture ReferenceObserver reference within a block, test may succeed false-positively.
If you are really afraid of it, you can add for instance an extra guard to a ReferenceObserver implementation and let it notify about deinit only after an explicit point in a test or manager reference lifetime by introducing extra scope using do{}. I would rather suggest to stick with a good-practice approach rather than adding an extra code. If you are interested, sample test with a safe-guard is available in this gist.

Summary

Presented solution suits very well for verifying “not-called” scenario that is fast and reliable. You may also find it useful to check whether something still keeps a reference to a block (with no reason) and potentially introduces memory leak.

Within couple of lines required to implement ReferenceObserver, you are finally sure that a given block will never be called without any delay. So now you can add this technique into you developer toolbox and protect yourself from unexpected Completion Inquisition.