Testing the properties of your code

When we write our suite test, we basically create example-based tests. We tend to create tests that our context (and biases) believe are good enough. Most of the time we only test the happy path and some corner cases. This is usually enough for the purpose of TDD, but testing in such a manner only demonstrates that our code works for a particular example or situation.

Although this kind of extended validation is usually associated with QA testing, wouldn’t it be much better if we could achieve it through our good-old test suite? Not as “In this particular situation this is true” but rather as “This property of the system is always true”. The latter is what is called property-based testing, as you test properties that should always hold.

Although we aren’t currently using this technique in Flywire, we think that in specific scenarios, it could provide us with a more robust test suite.

In the examples below we’ll be using the Rantly gem.

A sample scenario

At Flywire, we work with payments made by a student to a specific education institution. For us it’s natural to have in our system entities that represent each of these payments. Let’s say that we have the following definition for a payment:

class Payment
attr_reader :reference
attr_reader :amount_to_pay
attr_reader :amount_received
attr_reader :country_from
attr_reader :student_id
def initialize(reference, amount_to_pay, amount_received, country_from, student_id)
@reference = reference
@amount_to_pay = amount_to_pay
@amount_received = amount_received
@country_from = country_from
@student_id = student_id
end
end

This class defines the plain objects that exist in our system. Each one represents the information needed for a payment.

Let’s imagine that the product team created the following user story:

The operations team needs to identify payments based on the received amount, so they can take a closer look.

The rule they use is as following:

If the amount is greater than the amount received, the payment should be marked as under-payment.

Let’s create a test for this rule.

Example-based testing

This rule is really simple and straightforward. If the amount is greater than the amount received, then we have to mark the payment as under-payment.

In this test, we are following the AAA pattern. We first arrange the test with the payment we want to test, then we act producing the result that we want to test and finally we assert over it.

it “is under-payment” do
any_reference = 0
amount_to_pay = 100
amount_received = 20
any_country = “USA”
any_student_id = 10
payment = Payment.new(any_reference, amount_to_pay, amount_received,any_country, any_student_id)
  expect(is_under_payment(payment)).to be_truthy
end

This is the classical test that most people write. You choose an example, arrange that case and test it.

The correspondent implementation would be:

def is_under_payment(payment)
payment.amount_received < payment.amount_to_pay
end

From a QA point of view, an important case to notice is that if you change a value in the arrange part and the test still passes, this is a sign that there are scenarios that haven’t been taken into account.

Does this apply for our test? Can we find some values in the arrange part that once changed, still make the test pass? Yes, if we change the amount_received to 30, the test still passes.

Property-based testing

In our case, the particular set of data is the set of all the payments that satisfies the condition. If we could test our code against all the payments in the set, we could assert that the code works in every situation.

Let’s define a property instead of a fact.

it “is under-payment” do
property_of {
under_payments
}.check { |payment|
expect(is_under_payment(payment)).to be_truthy
}
end

With this implementation, we are expressing the following statement:

For all the payments that are under payment, the is_under_payment method will always be true.

And the result is:

……….
success: 100 tests
.
Finished in 0.00395 seconds (files took 0.08083 seconds to load)
1 example, 0 failures

What Rantly is doing, is testing that the check part passes for all (100 by default) cases in under_payments.

class Rantly
def countries
Rantly { choose(“USA”, “ES”, “FR”)
end

  def under_payments
amount_to_pay = float
amount_received = float

    guard amount_to_pay >= 0
guard amount_received >= 0
guard amount_to_pay > amount_received
    Payment.new(integer, amount_to_pay, amount_received, countries, integer)
end
end

We have to generate a set of cases that will be tested. For that purpose, Rantly will provide us a series of helpful primitives.

In the test, we have expressed that it should pass all of the cases in under_payments. In the class Rantly, we have defined a new method under_payments and this is the one that is called every time that Rantly needs to test a new case.

In the method under_payments we ask Rantly to generate two random float numbers for the amounts. Then we filter out the cases when the condition for an under-payment are not met. If the condition is met, we create a new Payment and test it.

This is way better than the example approach, as in this case we are expressing that the predicate is true in more than one case.

The more cases we use, the more stressed our code will be.

Conclusion

In this post we’ve seen that a test expressing a fact is in reality a more general case, as you could be missing many more scenarios. We have arrived to express a property of our system (All the payments that are under-payment should return true to the predicate).

As this is a property of our system, it must always hold. As we saw earlier, we can’t always test all the cases. In other words, they could be infinite, meaning that we haven’t proved our property but we can stress our code as much as necessary according to each situation.

This technique could also be used to discover system properties. Imagine that you have a code base that doesn’t have these kind of tests, but by reading the definition you can infer some property of the system.

It’s easy to define these properties as tests and then stress the code with cases and check if it breaks by finding a counter-example.

In sum, property-based testing offers the best of both TDD and QA-style testing. It helps you drive the development and for a little more effort, you obtain a significantly more robust test suite covering more cases.

Post written by: Javier Onielfa
Javier is a software developer at Flywire. If you have any questions, feel free to reach out to him on Twitter (@onielfadev).