Unit testing with the Hypothesis library

Tien-Duc Cao
Jun 4 · 4 min read

Introduction

Unit testing is a software testing method that allows any person involved in coding to validate that a given block of code performs as expected. With the traditional testing workflow, we need to specify precisely the inputs to test and therefore, if we choose too simple or easy inputs, then the test only covers those and isn’t a reliable tool to assess the quality of the code in case of a more complex or unexpected input.

The Hypothesis library allows us to describe how the inputs look like instead of specifying what they are. Then it generates many test cases with inputs corresponding to that description. The framework stops when it encounters a failed test case and raised an error specifying which input crashed the code.

A trivial example

  • Let’s write a function divide and a simple test function test_divide.
    The test case code runs successfully and it seems that our function works correctly.
def divide(x, y): 
return x / y
def test_divide():
divide(1, 2)
  • Now let’s rewrite the test_divide function with the Hypothesis framework and call this new function test_divide_hypothesis :
from hypothesis import given, settings
from hypothesis import strategies as st
@given(st.floats(), st.floats())
def test_divide_hypothesis(x, y):
divide(x, y)

The library will generate multiple inputs to test the code and in that case, it will report a ZeroDivisionError for the inputs (x = 0.0, y = 0.0), meaning we have to handle this case before going forward.

How to use the Hypothesis library

Writing a Hypothesis test is pretty straight-forward :

  • Decorate your test function with a decorator @given.
  • Describe the input data with strategies. Hypothesis has support for primitive data types and also provides the ability of building complex ones. A detailed reference can be found here.
  • Run the test with standard Python unittest or pytest.

Use case for Data Analytics / Machine Learning projects

Hypothesis supports numpy and pandas strategies.

To demonstrate this, we are going to test the function scale_min_max_v1.

def scale_min_max_v1(df):
for feature in df:
series = df[feature]
df[feature] = (series - series.min()) / (series.max() - series.min())
return df

Firstly we tell Hypothesis to produce a data frame with two columns “a” and “b” as float data type. We also add another decorator @settings to quickly rerun the failed test case later on.

def float_data_frame(features):
return data_frames(columns=[column(name=feature, dtype=np.float) for feature in features])
@given(float_data_frame(['a', 'b']))
@settings(print_blob=True)
def test_scale_min_max(df):
out = scale_min_max_v1(df)
if not out.empty:
for feature in out:
series = out[feature]
assert series.isnull().any() == False
assert np.isfinite(series).all() == True

After running the test (test_scale_min_max), Hypothesis reports the following error (data frame with values of 0 for both columns “a” and “b”) :

Falsifying example: test_scale_min_max(
df= a b
0 0.0 0.0,
)
You can reproduce this example by temporarily adding @reproduce_failure('5.16.0', b'AXicY2TAAQAAMgAC') as a decorator on your test case

We handle this error by returning 0 in the case of a 0 denominator. Let’s create a new function `scale_min_max_v2` to reflect those changes :

def scale_min_max_v2(df):
for feature in df:
series = df[feature]
denominator = series.max() - series.min()
if denominator == 0:
df[feature] = 0
else:
df[feature] = (series - series.min()) / (series.max() - series.min())
return df

With this updated function, we can check whether the failed test case is solved or not. To do that, the Hypothesis library gives us the decorator to add to test_scale_min_max :

@reproduce_failure('5.16.0', b'AXicY2TAAQAAMgAC')

This time, we receive a new message “hypothesis.errors.DidNotReproduce: Expected the test to raise an error, but it completed successfully.” which confirms the correctness of our fix!

Now let’s remove the reproduce_failure decorator and rerun the test. A new error is discovered with an infinity value (np.inf) for the column “b”.

Falsifying example: test_scale_min_max(
df= a b
0 0.0 inf,
)
You can reproduce this example by temporarily adding @reproduce_failure('5.16.0', b'AXicY2RABf8/QBkAEKkB8Q==') as a decorator on your test case

Similarly, we implement our new fix in a new function scale_min_max_v3 which replaces infinity values with 1e18 or -1e18.

def scale_min_max_v3(df):
df.replace(np.inf, 1e18, inplace=True)
df.replace(-np.inf, -1e18, inplace=True)
for feature in df:
series = df[feature]
denominator = series.max() - series.min()
if denominator == 0:
df[feature] = 0
else:
df[feature] = (series - series.min()) / (series.max() - series.min())
return df

After rerunning the test, another error is raised by the library (np.nan values in the column “b”).

Falsifying example: test_scale_min_max(
df= a b
0 0.0 NaN,
)
You can reproduce this example by temporarily adding @reproduce_failure('5.16.0', b'AXicY2RABf8/gClGBgAQqwHy') as a decorator on your test case

Let’s fix the error in scale_min_max_v4.

def scale_min_max_v4(df):
df.replace(np.inf, 1e18, inplace=True)
df.replace(-np.inf, -1e18, inplace=True)
df.replace(np.nan, 0, inplace=True)
for feature in df:
series = df[feature]
denominator = series.max() - series.min()
if denominator == 0:
df[feature] = 0
else:
df[feature] = (series - series.min()) / (series.max() - series.min())
return df

After removing the decorator reproduce_failure, the Hypothesis library has no more errors to report!
By default, Hypothesis generates 100 example test cases for each @given decorator (this can be increased with @settings(max_examples)).

Conclusion

The Hypothesis library helps us to discover hidden bugs in our code by automatically generate multiple and complex test cases (see testimonials here). As a result, it’s a little bit more time spent during the development phase but so much time saved if we had have to identify the root cause of a bug in production with unexpected inputs. Happy unit-testing!

Jellysmacklabs

Where all Jellysmack’s technology is made!