Writing Thorough Unit Tests

A pragmatic approach to improving your unit tests

As software developers we all know how important it is to unit test the code that we write. Some of us write tests that verify the happy paths only (bad) and others look for high code coverage numbers. But how many of you are actually writing thorough tests? Let’s take a look at what this means.

Characteristics of a good unit test

Thorough - Test normal & failure conditions, invalid inputs, and boundary conditions.

Repeatable - Tests must always give you the same results every time so avoid depending on things that might change such as an external server.

Focused - Tests should exercise one specific aspect of your code at a time. A failure in a unit test should lead you to an actual bug in your code.

Verifies Behavior - Tests should test behavior without making assumptions on implementations. Avoid over mocking with Mockito.

Fast - Tests should be fast since 70% of your tests should consist of unit tests.

Concise - Tests should be easy to understand and be a clear blueprint of what the code under test is doing.

Test 1: Not Repeatable

Above we see a test that is testing SomeClass#getCurrentMonthString. Way too often I see this mistake made during code reviews and it violates the repeatability characteristic. The reason is the default locale, timezone, and system time are being used. This means 1 month from now this test will return March instead of February which will instantly begin failing.

Let’s see how this same test looks with these issues fixed.

Here we can see we preset the timezone, locale, and timestamp so that we obtain consistent results.

Test 2: Not Fast + Repeatable + Thorough

This test just hurts me to look at. First off it’s doing network io in an actual unit test. Unit tests should be hermetic and mock network io so you mock responses and error conditions (401/503/200) to make sure your code handles every possibility.

It includes a sleep statement with some magical expectation that every time it’s run it will only take 10s to finish the network call. Please don’t use sleep statements EVER.

It’s also testing a method that performs some work asynchronously. Tests should always test against synchronous code.

Test 3: Not Thorough

Let’s assume we have a calculator class that provides addition, subtraction, multiplication, and division.

Here we have two test functions for addition and subtraction. Each test has verified the core function is working as intended.

While this will tick the checkbox for increased code coverage it’s clearly violating the thorough characteristic. You can’t just test the happy paths. You have to be extremely thorough and think of every possible way you can break your code and write tests accordingly. You should be adding two positive numbers, adding a positive & a negative number, a negative and a negative number, etc. You get the picture. Do not just test a single condition and move on. That’s not going to keep bugs from showing up in production code.

Test 4: Not Verifying Behavior

Here we have a custom EditText view that overrides the setText() function and prepends “vm-” to anything passed in. Let me explain why this is a bad test and why you should avoid Mockito.verify(). The common thinking here is that since EditText comes with Android then it has it’s own unit tests already so we don’t need to verify that it’s working.

Let’s say 6 months down the road someone wants to introduce a new parent class named CleanupEditText and one of its jobs is to strip out dashes from any input string. In this example with only a few lines of code it’s very easy to see where I’m going with this but imagine this class has 1000+ lines of code. Now the dev changes the parent class of MyEditText to this new class and runs all the tests to verify he hasn’t broken anything. The test that was written for MyEditText passes because all it is doing is verifying that the parent instance of setText() was called.

Now we’ve broken the expected behavior and have no way of knowing because the above test is not verifying behavior. It is only verifying a method is being called.

Here is that same test fixed so it verifies behavior.

How easy was that? Yet it’s so instinctive to reach for Mockito.verify() and abuse it without thinking of the possible consequences.

Test 5: Not Focused

Now let’s assume we have a class named Calculator again with basic mathematical operations supported.

Now someone writes a single test function called testOperations as shown below.

We should ensure we are identifying units of code and testing those separately otherwise you end up with an integration test as shown above. Remember, this article is about improving your unit tests and unit tests test business logic in isolation.

Here we have the same test broken up into 3 focused unit tests that are each testing only a single aspect of your code. The test function names have also been modified to more clearly explain what is being tested, with what inputs, and expected result.

Test 6: Not Concise + Thorough

Here we have a class named AnnualPayCalculator that computes an annual pay given hourly pay and total weekly hours worked.

Below you’ll see a test function that verifies the expected result is returned.

The main concern here is this test is not concise. It’s way too verbose and requires you to really read the test to understand what’s being tested.

Here is the same test written more concisely.

Test Naming

This brings us to our last characteristic of a good unit test that was not called out previously. Naming.

This is sure to be controversial because on one hand you are used to writing shorter function names in production code.

But, for test functions, there’s a crucial factor that changes the equation: you never write calls to test functions. A developer types out a test name exactly once — in the function signature. Given this, brevity still matters, but it matters less than in production code.

Whenever a test breaks, the test name is the first thing you see, so it should communicate as much as possible.

Format:

when_expected
when_with_expected

Bad examples:

testOnActivityCreated
testGetHashedAttachmentFilenames

Good examples:

viewCreated_listViewExists
getHashedAttachmentFilenames_withNullList_emptyStringReturned

Summary

• Don’t just write unit tests. Include integration and workflow where needed. Don’t waste time on these if they aren’t absolutely necessary to cover something that isn’t already being tested.
• Make sure your tests follow the characteristics of a good unit test
• Name your tests properly
• A failure in a test should immediately tell you exactly what is broken and under what condition
• Always strive to test functionality/behavior. Verifying a method is called only should be rarely used