What to Test When You’re Testing

Anatomy of a Unit Test

There are 2.0 kinds of people in the world, and because people are complex and computers are bad at handling floating points, each 1.999999998 of them brings their own infinite flavors of testing.

As with all things archaic, things were better back in the days. Someone would shout “Tests!” and a Pyramid would show up almost as magically (it was the ▲) as the real ones. And then there it was, the perfect embodiment of a society that loves its hierarchies:

The Testing Pyramid

We all know the basics, the lower we are at the pyramid the cheaper (in $$$ and in effort) it is to have a test fail and fix a fault/bug. And so the Unit Test was bestowed upon us — highly isolated, fast, cheap to run, with higher cardinality than its upper floor neighbors. The folks at the top are usually lower in numbers but make up for it by testing a deeper level of integration between the components and in case of an E2E (End-to-End Test) even subsystems.

Systematic Integration

One simply needs to <insert-search-engine-of-choice> "How to test ..." and they'll encounter a sea of information appealing to both individual taste and distaste. This article included. But why? Why can't we have nice consistent things?! Because reasons. Also circumstances. The Pyramid drives the base of everything holy, but individual situations drive its application. A Browser component may lean towards a certain strategy that might differ from that of a CLI, which in turn could be off-tangent (by gargantuan proportions) for an API Server. The list adds up and also includes infallible infrastructure kept together by sticks, gum, and divine bash scripts.

One fundamental thing most would agree to is that we need tests for a certain amount of confidence in our systems, software or otherwise. (Unless of course, you belong to the other infinite group of 0.9999997 opinions with a varying scale of resolve between "Tests are an overkill" and "Maybe just this one time I don't need them".)

Units of a Unit Test

Sometimes we encounter situations where there’s a thin line between an Integration and a Unit Test. But why is that line thin? Is it maybe that we’re looking at it from far away? Perhaps it helps to come a bit closer and examine it from where the line is actually looking broad enough.

In ideal situations, a good Unit Test looks like art and reads like poetry. It also fails with the grace of fallen meteorite — it tells you the impact it caused, exactly where it came from, and usually leads to a bunch of nerds grouped together trying to understand the situation and following up to rectify the damages. It’s glorious.

But how does one write such an idyllic test?

We start simple and discuss the features under titles that all sound like Bethesda’s video game titles, playing nicely with our aforementioned analogy.

Fallen Sky: Impact

A test fails, what’s the first thing to cross your mind?

“Dammit! Now I need to fix it on my Friday evening”

Great once that’s subsided…

“Okaay, what failed?”

An ideal Unit Test would tell you exactly that and it does so by two means:

1. Name
2. Assertion Failure

What’s In a Name?

Let’s assume a not so hypothetical scenario of MyVeryCoolAPI. Amazing Alice sits on her desk where she ponders why she can't afford that nice condo in Miami, while simultaneously wishing why she didn't pick a career in beekeeping. It seems far less stressful. But we digress, Alice is writing some code that registers a user:

func RegisterUser() {
    if model.AlreadyRegistered() {
        errorAlreadyRegistered
    }
    if model.FirstNameTooShort() {
        errorFieldTooShort
    }
    if model.PasswordInsecure() {
        errorPasswordInsecure
    }
}

And rightly so she proceeds to write some tests for it:

func TestRegisterUserFails() {
    ...
}

Let’s take a moment to reflect upon what could happen when the test fails? You see a message from your favorite test runner:

ERR! “TestRegisterUserFails” failed at ./src/user.code:42

Great! We now know that RegisterUser has a bug, but what is it? Guess time to open the code editor. Hope it has deeplink support.

It would’ve been ideal if the test itself told us what it was about, maybe something like:

func TestRegisterUserFailsIfPasswordInsecure() {
    ...
}

ERR! “TestRegisterUserFailsIfPasswordInsecure” failed at ./src/user.code:42

Looking at the error above, even in the CI/CD pipeline, we get an immediate context of what failed and where. One can immediately hit the ground running and start fixing the bug in that part of the code. Hopefully without creating more bugs.

“Written code is potential fodder for bugs. Don’t feed the bugs.”

Heeding our advice, Amazing Alice proceeds by renaming all tests that convey their intent. In process creating yet another unfortunate test:

func TestRegisteredUserHasWelcomeInviteCode {
    ...
    assert("WELCOME42" ==  user.InviteCode)
}

ERR! “TestRegisteredUserHasStandardInviteCode” failed at ./src/user.code:42

Expected: true
Actual : false

Upon encountering this error, we know immediately what failed and where. However, most test runners give out a meaningful diff between the expected and the actual value and it's definitely in our best interests to leverage it. Standing on the shoulders of giants and all that hipster talk.

Reading the error, the test name gives us the following information:
“test failed because registered user had a wrong invite code” But the Expected and Actual give us next to no information. Something was supposed to be true but was instead false. Might we remind ourselves that we are in the process of fixing a bug and not caught between a rendition of Shakespeare's Hamlet questioning our binary choices. It would've been colossally helpful if we in fact knew what code was expected and what we got.

Enter: Assertions. Most testing libraries would have some form of basic support for asserting/comparing two values (some prefer it as Expected and an Actual value). Often simply changing the assertion leads to a clearer context around a failure:

func TestRegisteredUserHasWelcomeInviteCode {
    ...
    assertEqual("WELCOME42", user.InviteCode)
}

ERR! “TestRegisteredUserHasStandardInviteCode” failed at ./src/user.code:42

Expected: WELCOME42
Actual : WELCOME42\n

“Oh my Lord!”, announces an anguished Amazing Alice (oblivious to the alliteration), for she had forgotten to sanitize the form input for whitespace. She looks around the room.

“One who has not sinned may cast the first stone.”

We’ve all been there. Join the club Amazing Alice. Here’s your 1-Day chip.

Fallen Sky: Explorer

Another thing that helps in identifying a failed test is what the test does functionally.

Having joined our cool club, Amazing Alice hands over the codebase to Basic Bob. Wait… why Basic? Well it’s not like Bob’s a bad programmer, he just happens to love the language BASIC. The name just stuck.

Hanging out at the water cooler Basic Bob has a Eureka! moment and off he runs to write the following test:

func TestRegisteredUserIsValid {
    input := UserInput {
        Name: "john",
        Email: "john@doe.com",
        Phone: "1800-000-000",
        WelcomeCode: "WELCOME42\n",
        ...
    }
    user := model.RegisterUser(input)
    assertEqual(db.GetWelcomeCode(), user.GetRegisteredWelcomeCode())
    assertEqual("john", user.GetName())
    assertTrue(user.IsValidEmail())
    assertTrue(user.IsValidPhone())
}

The dense collection of asserts makes Archimedes proud. And Basic Bob as well. Until, that is, one fine day where the test fails:

Given the following error:

ERR! “TestRegisteredUserIsValid” failed at ./src/user.code:42

Expected: WELCOME42
Actual : WELCOME42\n

Great, thinks Basic Bob, must be the Welcome Code again and proceeds to fix it.

And then a month later, yet another failure:

ERR! “TestRegisteredUserIsValid” failed at ./src/user.code:42

Expected: true
Actual : false

Having recently returned from his vacation in the Caribbean, Basic Bob has forgotten all things code, especially the ones he wrote himself. Thankfully the assertion failure has a line number and Basic Bob is quick to observe it and denotes it as invalid email failure.

Cut to another month and the Product Office® comes with the requirement that name MUST be Capitalized. Faced with this information and a choice, our hero decides to fix the test:

func TestRegisteredUserIsValid {
    input := UserInput {
        Name: "john",
        Email: "john@doe.com",
        Phone: "1800-000-000",
        WelcomeCode: "WELCOME42\n",
        ...
    }
    user := model.RegisterUser(input)
    assertEqual(db.GetWelcomeCode(), user.GetRegisteredWelcomeCode())
    assertEqual("John", user.GetName())
    assertTrue(user.IsValidEmail())
    assertTrue(user.IsValidPhone())
}

Aweso….oh wait it fails out of the blue the week after:

ERR!“TestRegisteredUserIsValid” failed at ./src/user.code:42

Expected: John
Actual : john

More exploration on why it failed is underway. Visit line number 42. Rinse and Repeat. Basic Bob enjoys his little field trips down the code lane.

The main reason that this test sends Basic Bob to numerous explorations is purely because it tests too much. Every time a specific case fails the entirety of the assertions must be checked and explored to figure out what failed. But more importantly, the test fails to capture the Product Office® ‘s requirement on why the input john was invalid - It's not Capitalized.

A better approach would’ve been to separate out the tests to their own units, adapting the test cases as necessary:

func TestRegisteredUserHasValidWelcomeCode {
    input := UserInput {
        WelcomeCode: "WELCOME42\n",
        ...
    }
    user := model.RegisterUser(input)
    assertEqual(db.GetWelcomeCode(), user.GetRegisteredWelcomeCode())
}
func TestRegisteredUserHasCapitalizedName {
    input := UserInput {
        Name: "john",
        ...
    }
    user := model.RegisterUser(input)
    assertEqual("John", user.GetName())
}
func TestRegisteredUserHasValidEmail {
    input := UserInput {
        Email: "john@doe.com",
        ...
    }
    user := model.RegisterUser(input)
    assertTrue(user.IsValidEmail())
}
func TestRegisteredUserHasValidPhone {
    input := UserInput {
        Phone: "1800-000-000",
        ...
    }
    user := model.RegisterUser(input)
    assertTrue(user.IsValidPhone())
}

Splitting up the test cases not only makes failures more explicit, but also makes them future proof to update individual tests with additional context as things evolve. Which will happen. No exceptions.

Having split the tests up, Basic Bob returns to the water cooler, waiting for someone to join him so they can re-engage in their casual non-productive workplace banter.

Fallen Sky: Seer

Esoteric Eve loves subscribing to Repositories. It’s not like she enjoys eavesdropping on the notifications from Alice and Bob, she’s just really, and I mean reaally good at doing Code Reviews.

People seek her out when they wish to know doth their code fail hereafter. For the eyes of mere mortals can not see past their own f̶o̶l̶l̶y̶. code.

But as much as Esoteric Eve enjoys reading code to spot potential pitfalls, she has a certain knack for getting tricked by obscure code:

func TestRegisteredUserHasNumberCorrectlyFormatted {
    u := UserInput {
        Phone: "1800123457",
        ...
    }
    assertEqual(formatNumber(u.Phone), model.RegisterUser(u).GetPhone())
}

By Golly! exclaims Esoteric Eve. She knows exactly what the test should be doing, but how is it doing it? She stumbles in the darkness looking for a glimpse of anything that feels familiar: an input, an expectation, a unit under test, the actual output, the strong smell of Coffee, God, Netflix, anything.

But only once do the heavens part and the holy light shines upon the code, can she finally decipher Da Vinci’s code. Exasperated, Esoteric Eve endearingly thinks to herself — how could a good feedback on the PR look like? — what’s important to her? She brings forth the following commandments:

Thou shalt clearly identify,

• The input parameters for the test
• The unit under test
• The expectation that should be asserted
• The actual output that came out of the unit under test

And as she imagines the perfect test, ASCII characters start materializing around her. She still had to write them down via her keyboard though:

func TestRegisteredUserHasNumberCorrectlyFormatted {
    input := UserInput {
        Phone: "1800123457",
        ...
    }
    user := model.RegisterUser(u)
    expected := formatNumber(input.Phone)
    actual := user.GetPhone()
    assertEqual(expected, actual)
}

The above generic template embodies good practices when writing a unit test that is meant for reading. Clearly identifiable inputs, expectations, actual outputs, limited asserts and limited function calls. Some form of inlining is totally acceptable within specific cases, but as with all things, moderation should be the key. Avoid obfuscating code for your own peers for no reason (potential valid reasons: trying to get a backdoor through a code review).

Just about to press the Enter key, abruptly comes a premonition to our Esoteric Eve and she screams out loud - in her head - "Generated Expectations!"

Wow, that was a close one. There was, incidentally, a huge problem with the test as it stood. The thing with expectations is that they should generally (also ideally) be static and thus as a general rule, should be typed out. Imagine that the code in question itself is using the formatNumber() internally. By saving ourselves some time by generating expected := formatNumber(input.Phone) we effectively gave the bug a Get Out of Jail Free card. If there would occur a bug in the formatNumber() function we would've never caught it with this code! Yikes.

So, having recovered from the premonition, our Esoteric Eve frantically makes the edit:

func TestRegisteredUserHasNumberCorrectlyFormatted {
    input := UserInput {
        Phone: "1800123457",
        ...
    }
    user := model.RegisterUser(u)
    expected := "1800-123-457"
    actual := user.GetPhone()
    assertEqual(expected, actual)
}

Feeling relived, she thanks her stars, not just for the PR-related premonitions but also the stocks she just bought last weekend. The world shall never know.

End Thoughts

Having role-played through the above game titles, it’s quite important to sit down and recollect our experiences. Too Much Information. What’s the tl;dr dude?

The tl;dr — well it’s situational, complicated and personal. All the above cases are not gospel, yet shed light on common pitfalls one can get themselves into. But the tests are so small, why bother?. Because habits develop. One can on many instances identify the same patterns in bigger codebases and their related tests. Even when the tests start small, people come and go, and the tests evolve — and they evolve into a behemoth that could haunt either of Alice, Bob or Eve. We don’t want that. Nobody wants that. Especially not those Three.

So, having decided that we don’t wish to cause anyone discomfort, the questions one can ask themselves, and then decide on their situation accordingly, could casually be summarized as:

• Can a different colleague identify my (singular) Unit under test?
• Do I have a high amount of assertions that defy rational human logic?
• Can one uniquely identify my input, expectation and actual output?
• Provided this test fails, do I and my future counterparts know where to look for answers?
• Upon failure, does the error message successfully relay the behavior that was violated?
• Is my test using generated expectations?
• Am I causing people unnecessary discomfort?

These self-questions are certainly not exhaustive but definitely signify a start in our collective journey to the land of glorious tests. But most importantly, one should truly enjoy writing great Unit Tests. And after having finished writing each one of them, pause in awe to exclaim to their nearest focused colleague — Darn, this really looks like art, reads like poetry and fails with the grace of fallen meteorite!