Use Test Coverage to Defend against Typos

When you need to type dozens of if statements, do you trust your body to make zero mistakes? If you don’t, perhaps this helps.

As W. Edwards Deming’s famous quote goes,

In God we trust. All others must bring data.

At Xendit, we believe in data-driven decision-making, for example, one of our fraud detection products quantifies the integrity of a transaction by examining multiple data points. In code, this looks like

only with about ten times more data points.

Excellent, clear, and on point. What could go wrong? 😆

Quality Assurance Inferno

The problem is, I do not trust my hands not to make typo errors, hence testing is pivotal; however, when you have so many conditional branches, comprehensive testing becomes… impossible. Math time.

Let’s assume we are dealing with 50 data points, all independent and binary (e.g., can only be “true”/“false”, or “nil”/“not nil”). There are 2⁵⁰, about 1 quadrillion, combinations of input! Putting this into perspective:

• If one case takes 20 seconds to write, I need 714 million years to finish it, even if I don’t eat or sleep 💀;
• If one case is 100 bytes long, this test is 100 petabytes big (good luck git cloning 😅).

A Conventional Compromise

A colleague shared a testing technique where each test case triggers one more condition than the previous one. In other words, test case #0 triggers none of the conditions, #1 triggers nothing but the first condition, #2 first two conditions, and so on.

(Illustrated at https://go.dev/play/p/FMp0IPCS5mX)

This looks promising, however, I’m still worried because if I make a typo in the test, say mixing IsRelatedToNegativeEvent and IsPhoneLeaked which happen to have the same weight now, one branch would be left untested. Besides, with n conditions, the number of lines of code will be of magnitude O(n²) (roughly, test case #1 has 0 lines, test case #2 has 1 line, #3 2 lines, and so on), adding to the chance of typo.

Make Use of Test Coverage

My paranoia above actually inspired me. To leave no stone unturned, I could make 50 test cases, each only triggering one condition; then, I could check the test coverage to see if I missed any branch!

For your information, Go provides a convenient tool to view test coverage. To use it, simply add -coverprofile flag, providing a file path to your go test command, and view it with go tool cover command, like

go test -coverprofile=cp.out ./...
go tool cover -html=cp.out

With a test like

(Notice the last test case has a typo),

the output looks like

(of course, you might not want to check in this cp.out file to source control.)

Compared to the conventional compromise, this will go wrong only if both the implementation and the test have the exact same typo error. Because of this, I shall type the tests and the implementation separately.

An Aesthetic (but not recommended) Refactor

Perhaps the multiple if statements would be frowned upon by human reviewers or your linter might complain about its cognitive complexity. One way to solve that without compromising robustness could be to define weightFunc like func(response) int, extracting each conditional statement into a separate function, then looping through them.

(Illustrated at https://go.dev/play/p/yWFTtYVl6Id)

Personally, I don’t recommend it because there are more to type, hence increasing the chance of introducing typo errors, and it does not make the tests any easier — for this, the quality depends much less on technical complexity and more on human errors.

In summary, if you have such repetitive conditional statements, consider using one test case for each branch, and use test coverage to help examine if all are covered. With tests like https://go.dev/play/p/JtaUp4RVnkP, the test coverage would look like

Thank you for reading, and I hope this helps!