Beyond unit tests: an intro to property and law testing in Scala
I have been using ScalaCheck testing library for at least 2 years now. It allows you to take your unit tests to the next level.
- You can do Property-based testing by generating a lot of tests with random data and asserting properties on your functions. A simple code example is described below.
- You can do Law testing that is even more powerful and allows you to check mathematical properties on your types.
Property-based testing
Here is our beloved User
data type:
case class User(name: String, age: Int)
And a random User
generator:
import org.scalacheck.{ Gen, Arbitrary }
import Arbitrary.arbitraryimplicit val randomUser: Arbitrary[User] = Arbitrary(for {
randomName <- Gen.alphaStr
randomAge <- Gen.choose(0,80)
} yield User(randomName, randomAge))
We can now generate a User
like this:
scala> randomUser.arbitrary.sample
res0: Option[User] = Some(User(OtwlaaxGbmdhuorlmgvXitbmGfbgetm,22))
Let’s define some functions on the User
:
def isAdult: User => Boolean = _.age >= 18
def isAllowedToDrink : User => Boolean = _.age >= 21
Let’s claim that:
All adults are allowed to drink.
Can we somehow prove this? Is this correct for all users?
This is where property testing comes to the rescue. It allows us not to write specific unit-tests. Here they would be:
- 18-year-olds are not allowed to drink
- 19-year-olds are not allowed to drink
- 20-year-olds are not allowed to drink
All of these statements can be replaced by a single property check:
import org.scalacheck.Prop.forAllval allAdultsCanDrink = forAll { u: User =>
if(isAdult(u)) isAllowedToDrink(u) else true }
Let’s run it:
scala> allAdultsCanDrink.check()
! Falsified after 0 passed tests.
> ARG_0: User(,19)
It fails as expected for a 19-year-old.
Property testing is awesome for a few reasons:
- Saves time by writing less specific tests
- Finds new use cases generated by Scala check that you forgot to handle
- Forces you think in a more general way
- Gives you more confidence for refactoring than conventional unit tests
Law testing
It gets better: let’s take it to the next level and define an Ordering between Users:
import scala.math.Orderingimplicit val userOrdering: Ordering[User] = Ordering.by(_.age)
We want to make sure that we didn't forget any edge cases and that we defined our order properly. This property has a name, and it’s called a total order. It needs to holds for the following properties:
- Totality
- Antisymmetry
- Transitivity
Can we somehow prove this? Is this correct for all users?
This is possible without writing a single test!
We use cats-laws
library to define the laws we want to test on the ordering we defined:
import cats.kernel.laws.discipline.OrderTests
import cats._
import org.scalatest.FunSuite
import org.typelevel.discipline.scalatest.Disciplineimport org.scalacheck.ScalacheckShapeless._class UserOrderSpec extends FunSuite with Discipline {
//needed boilerplate to satisfy the dependencies of the framework
implicit def eqUser[A: Eq]: Eq[Option[User]] = Eq.fromUniversalEquals//convert our standard ordering to a `cats` order
implicit val catsUserOrder: Order[User] = Order.fromOrdering(userOrdering)
//check all mathematical properties on our ordering
checkAll("User", OrderTests[User].order)}
Let’s run it:
scala> new UserOrderSpec().execute()
UserOrderSpec:
- User.order.antisymmetry *** FAILED ***
GeneratorDrivenPropertyCheckFailedException was thrown during property evaluation.
(Discipline.scala:14)
Falsified after 1 successful property evaluations.
Location: (Discipline.scala:14)
Occurred when passed generated values (
arg0 = User(h,17),
arg1 = User(edsb,17),
arg2 = org.scalacheck.GenArities$$Lambda$2739/1277317528@41d7b4cf
)
Label of failing property:
Expected: true
Received: false
- User.order.compare
- User.order.gt
- User.order.gteqv
- User.order.lt
- User.order.max
- User.order.min
- User.order.partialCompare
- User.order.pmax
- User.order.pmin
- User.order.reflexitivity
- User.order.reflexitivity gt
- User.order.reflexitivity lt
- User.order.symmetry
- User.order.totality
- User.order.transitivity
Sure enough, it fails on the antisymmetry law! Same age and different names are not supposed to be equals. We forgot to use the name in our original Ordering
, so let's fix it and rerun the laws:
implicit val userOrdering: Ordering[User] = Ordering.by( u => (u.age, u.name))scala> new UserOrderSpec().execute()
UserOrderSpec:
- User.order.antisymmetry
- User.order.compare
- User.order.gt
- User.order.gteqv
- User.order.lt
- User.order.max
- User.order.min
- User.order.partialCompare
- User.order.pmax
- User.order.pmin
- User.order.reflexitivity
- User.order.reflexitivity gt
- User.order.reflexitivity lt
- User.order.symmetry
- User.order.totality
- User.order.transitivity
And now it passes :)
If you are wondering what can you test besides Order
s, go check out the docs here: https://typelevel.org/cats/typeclasses/lawtesting.html
Summary
- Property tests are more powerful than unit tests. They allow us to define properties on functions and generate a large number of tests using random data generators.
- Law testing takes it to the next level and uses the mathematical properties of structures like
Order
to generate the properties and the tests. - Next time you define an ordering and wonder if it’s well-defined, go ahead and run the laws on it!