Mocking Can Lead To False Positives 🔊

The danger of using primitives to compare test values

Fagner Brack
Jun 9, 2016 · 4 min read
A man, tying his shoes with a fake arm
Listen to the audio version!

Let's say that we have a function "add index to database" that accepts an index which is a primitive. The purpose of the function is to run a complex operation before calling another function inside it called "database persist", passing that same index:

Let's say, for the sake of the argument, that:

  1. This is a piece of a legacy code with low test coverage. We need to test as much as possible to reduce the chances of breaking how that function currently behaves.
  2. For the reason above, we can't do Test-First.
  3. We want to test that the "database persist" function will be called with the index input we provide.

Given those constraints, we need to mock the "database persist" function and check if that function will be properly called with the value we want:

Everything looks fine, except for the fact we introduced a potential false positive in the test code.

What if inside the body of the function there's a change that reassigns the index argument to a value of 5 that is coincidentally the same value we are using for the "fake index" input in the test?

In this case, the test will pass because it's not really checking if the "database persist" function is being called with the index input we provided, but rather that it's being called with 5, assuming an input of 5.

In the future, if somebody overrides the index variable (which is assumed to be an invariant throughout the execution of the "add index to database" function), then there will be a non-deterministic test that will keep passing even if the mocked function is not really being called with the input.

That is not a very useful test.

This problem happens because when we test the arguments of the mocked "database persist" function we are comparing the argument value, instead of comparing the reference.

Testing that the inner mocked function is called with the primitive input of the parent function can lead to false positives because we are comparing the value, not the reference

This seems obvious in hindsight, but it can easily pass unseen, even in code review.

One solution for this problem is, instead of passing a primitive as the "fake input" in the test, we pass the instance of some dummy object so that when we compare with the "fake input" we can be sure that we are comparing with the actual reference, instead of comparing with the value.

For this solution, it doesn’t matter which type the "add index to database" function accepts. We can pass an Object Literal just for the sake of holding the reference in the variable so that we can have a proper deterministic test:

Passing a dummy Object Literal will be more robust and future-proof. But only if we can duck type the interface of the original “index” argument to an Object Literal. The "add index to database" function might be using the index in a way it forces a caller to call it with a number argument, like when it uses the input for arithmetic operations.

If you don’t understand the meaning of “interface” in this context, take a look at this post.

If there are arithmetic operations like index + index, those cannot be duck typed to an Object Literal and therefore we will need to use a new instance that provides the same interface of a number, like new Number():

That will work, because now we are creating a specific instance, and checking against that instance, instead of against the primitive value.

That will also allow the code to treat the input as a primitive for most use cases throughout the test, so that if the arithmetic operations change in the future, then the test will still pass legitimately.

The instance approach works perfectly when we can substitute the primitive, but it doesn't work when we are passing values such as null or undefined, which have no equivalent way to be passed by reference.

Leveraging duck typing to pass by reference instead of passing by value doesn't work when we pass something that doesn't have a reference equivalent that can be duck typed

The example below shows a false positive being introduced when we test the “add index to database” function with an undefined input:

In that case, a solution for the false positive can be something like property-based generative testing. It will test for the mocking call property using many samples instead of relying on a single one. Another solution can be triangulation.

Unfortunately, that means adding more abstractions into your tests and relying on statistics, but considering the weight of the benefits and the low likelihood of your code reassigning the same types that are being generated, it might not be all that bad.

Sometimes mocking in unit tests can be tricky. Understanding exactly what we want to test and how the language comparison system works is a must to avoid some dangerous and subtle traps.

Generative testing and triangulation can be an alternative solution that use statistics to improve the determinism of a mock call, but as most of the things in software development, there are no silver bullets.

Maybe what you can do is not to use mock at all.

This article became a talk: Mocking And False Positives. The slides have other examples and more details.


Thanks for reading. If you have some feedback, reach out to me on Twitter, Facebook or Github.

Fagner Brack

Written by

I believe ideas should be open and free. This is a non-profit initiative to write about fundamentals you won’t find anywhere else. ~7 min post every few weeks.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade