Choosing and managing examples in your tests

It is very important to wisely choose the examples we use to write our tests so that they are useful and meaningful. In this article, I will explain how to achieve that using basic techniques.

Choosing examples for tests

On the other hand, it is frequent to need the same examples in different tests. When we work with an object-oriented approach, it is easy to refer to certain objects according to their properties. For example, you’ll want to work with an example of a logged user, or of a booked appointment, or of a canceled order, etc., in different scenarios across the test suite. In those cases, you can copy-paste them, but there is a better solution for that - one that ensures that you have a coherent set of examples, with a unique creation point for them: object mothers.

So, let’s go first with how to choose examples.

Choosing examples

Unique discrete values: test them all

When we have a limited number of possible discrete cases, we can simply test all of them.

Example: applying some kind of promotional code to a price. Suppose that there are only three promotional code types: low, medium, and high. They provide a discount of 10%, 20%, and 30%, respectively. So we write a test using each one.

Test all cases if you have a limited set of them

Values that you can group by some criteria: equivalence class partitioning

It is frequent to have infinite values for a variable, but they can be easily classified in categories based on a property of the data that is relevant for the test. These categories are equivalence classes and all of their members are equally representatives of the class.

You are not going to try all the possible values, so you select one representative from each category, and that’s enough.

Let’s consider the Leap Year problem. There are the following categories:

• Common years: those which number is not divisible by four.
• Leap years: those which number is divisible by four, but not by 100.
• Special common years: those which number is divisible by 100.
• Special leap years: those which number is divisible by 400.

You only need to test an example of each category

So, we only will need four examples to prove all of the possible scenarios. Well… maybe we could add a special scenario for dates before 1582, given it marks the invention of the Gregorian calendar, but I think that this is not relevant for most current businesses.

We can classify values, but we care about what happens in the boundaries between categories: boundary value analysis

Some variables can accept continuous values; we can partition this continuous into categories. Nevertheless, we are worried about how the behaviour applies to the values in the boundaries.

In other terms: categories are intervals of values ​​whose extremes can be open or closed. Open extremes are excluded from the interval, while closed extremes are included.

We can consider the Museum Ticket Price problem. It’s common practice to segment the price of these tickets based on age. Categories are defined by age intervals, like this:

• Toddlers: Less than 3 years old.
• Children: Between 3 and 12 years old.
• General: Between 12 and 65 years old.
• Retired: More than 65 years old.

The point to consider here is in which category is included each boundary. We need a precise definition from business people. For example:

• Toddlers: Less than 3 years old.
• Children: More or equal than 3 and less than 12 years old.
• General: More or equal than 12 and less than 65 years old.
• Retired: More than 65 years old.

So: how many examples do we need? If we want to be sure about the boundaries we would need to pick two values in each category - which are the values in both extremes. For the categories with only one boundary, we can pick only one example.

• Toddlers: 2
• Children: 3, 11
• General: 12, 64
• Retired: 65

This way, we will be well served with six examples. Note that each example is a good representative of its category.

We use boundary analysis when it is important to be sure about the behaviour across different categories

We need to create scenarios based on several variables: decision table

If we need to combine several variables to create a test scenario we apply a combinatory strategy.

First, we choose examples for each variable, applying the equivalence class partitioning or the boundary value analysis strategy.

Second, we combine all using a combinatory tree and represent the combinations in a table.

Decision tables can be huge

We are not sure about the values that we need to try: basic path

The previous techniques are used for black-box testing. This is the kind of test that we write considering the subject under test as a black-box, for which we don’t know anything about how it is implemented.

Sometimes, we don’t have a clue about the values that we could use for testing, but we can examine the algorithm in order to ensure that all the possible branches of the flow are executed. With this knowledge, we can define the different scenarios. Usually, the different execution paths are controlled by a variety of variables, so the black-box techniques don’t fit well.

This type of testing is what we refer to as white-box testing.

We can see some of the possible paths of this algorithm.

We can use graphical code coverage tools to discover the paths not tested yet.

Managing examples in tests

Making our selection of examples understandable in the tests can be a tricky problem. When we write the tests we know why we choose certain values as examples for a concrete test, but another developer that is not familiar with the subject can feel a bit lost.

Test examples fall easily in the Magic numbers code smell: arbitrary values in the code of the test without meaning nor explanation.

Name your examples by applying the introduce constant refactor (or introduce variable)

The best way to solve this problem is by introducing constants or variables with meaningful names instead of the raw values. This way, the reader can understand what’s going on without the need to know the exact values, but they can always access them.

Using constants or variables is up to you, and can be determined by the programming language. It depends on the scope in which you need to use the example.

Empty description is represented with a constant in the test that explains the meaning of the empty string

Apply extract method when it is complex to create the example

Naming examples using constants or variables works fine for scalar values and simple objects, but if creating an example requires some complexity it becomes easy to entangle the code, making it difficult to understand.

In those cases, we can extract the complexity to a factory method in the test case, so instantiating an example becomes a one-liner. You can parametrize the method to be able to customize the example for different cases.

In the following example, you can see how we extract the instantiation of the Request object to a private method, making it easier to create all examples that we need.

Maybe you need to use builders or factories for this.

Apply the object mother pattern if you want to use the same examples in different tests

Many times you will want to use the same example across different unrelated tests, because you would like to have a way to reuse or share it in some way.

To achieve that, you can apply the Object Mother pattern. An Object Mother is similar to a limited factory that can create valid examples of certain objects. Usually, they are completely configured, but you can allow some parametrization - you could say that they provide archetypical examples of objects.

Here is an example. I’d prefer to name these objects as Examples, but the pattern is the same.

Remember that Object Mothers are not builders. A builder is an object that knows how to instantiate other objects that have complex construction. An Object Mother is only concerned about a very reduced set of parameters, keeping the rest constant, default, or even random. So, if your Object Mothers need to be builders, that’s an indicator that you are trying to test too much at once or that the object is too simple, as happens in our example.

Some tips with Object Mothers.

Allow them to be used statically; granting a unique place to gather examples of a class. It’s safe to do so as they only provide this creational behaviour and they will be only used in tests.

Name methods meaningfully. Each factory method must describe the type of example it creates, so the team will be able to talk and reason about them. For example, I usually have a dummy method, of which returns an arbitrary configured object for use in examples in which the result doesn’t depend on particular properties of the object.

Another example: you can have a ContractMother::signed to provide a Contract object that is signed to compare against an unsigned one.

This allows the team to have conversations about the tests and their examples.

Limit parametrization to only one or two variables, if needed by the tests. Keep constant or randomize the rest of the required parameters; if there are only a few values, create a factory method specific for each one.

Stochastic examples?

You can think that randomness and tests don’t mix well. This is generally true, but you can use some randomness to increase the reliability and trust in your test suites.

How can you do that? By introducing random examples in the tests instead of using fixed ones. The point is to not test based on single values for a category, but instead, you try randomly with a lot of values that belong to the category.

The random method in the following snippet generates a completely random Task object every time it is called. Using these examples your test will run with a completely new set of examples, and the code under test should be able to manage them properly:

So, tests using this approach could fail at a random execution but this is the intended behaviour. Your task is to avoid that a failure can happen even when we are providing random examples, something that resembles real production situations.

Summarizing

There are several techniques that you can use to choose examples for tests in a systematized way:

• The limited set of examples
• Equivalence class partitioning
• Boundary analysis
• Decision table
• Basic path

Also, you can manage all those examples in your test by using simple practices:

• Use constants and variables to give them meaning
• Extract creation of examples to private methods
• Use the Object Mother pattern to have a limited set of examples, for using in tests and to reason about

Happy testing!