Property Based Testing: ScalaTest + ScalaCheck

Property Based Testing is an another strategy or philosophy of testing software’s

A property can be any behavioural characteristic of a method or object which should hold true on any fuzzy situations.

Property Based Testing derived from Functional Programming community: QuickCheck in Haskell

Definition:

Property based testing is the construction of tests such that, when these tests are fuzzed, failures in the test reveal problems with the system under test that could not have been revealed by direct fuzzing of that system.

Let’s quickly see how ScalaTest and ScalaCheck works

ScalaTest:

ScalaTest is concise, easy-to-read testing framework developed to write quick test cases and express them in near plain text

ScalaCheck:

ScalaCheck is the scala framework for Property Based Testing. It has a good integration with ScalaTest as well.

Let’s jump to code and see how simple they are,

Maven Dependency:

<dependency>
    <groupId>org.scalatest</groupId>
    <artifactId>scalatest_2.11</artifactId>
    <version>3.0.8</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.scalacheck</groupId>
    <artifactId>scalacheck_2.11</artifactId>
    <version>1.14.0</version>
    <scope>test</scope>
</dependency>

Assume we have two method’s ,

• div — Dividing two numbers
• append — Append an Int to the end of the list

object Main {

  def div(n: Int, d: Int): Double = {
    require(d != 0, "Denominator should not be 0")
    n / d
  }

  def append(list: List[Int], postfix: Int): List[Int] = { 
    list :+ postfix 
  }
}

In ScalaTest, we have two styles of writing ScalaCheck —

1. ScalaCheck style
2. Using ScalaTest Matchers [We will use this]

I will explain how we test with and without ScalaCheck to see the difference,

Without ScalaCheck:

"div function" should "return expected values" in {
  div(4, 2) shouldBe 2
}
"append function" should "return only one element when added in empty list" in {
  val emptyList = List.empty[Int]

  append(emptyList, 1) should have size 1
}

The test uses Matchers trait to use shouldBe clause in defining the assertion. ScalaTest supports many similar assertion shorthands which makes it readable and concise.

With ScalaCheck:

"div function" should "return expected values" in {
   forAll(arbitraryInts, nonZeroInts) { (n: Int, d: Int) =>
     div(n, d) shouldBe n / d
   }
}
"append function" should "return only one element when added in empty list" in {
  val emptyList = List.empty[Int]

  forAll(arbitraryInts) { n: Int =>
    append(emptyList, n) should have size 1
  }
}

Defining property in ScalaCheck is pretty simple: In the example above we are testing the property of div and append function given some fuzzy or random inputs and still they produce result as expected.

forAll method is used to provide the inputs — example above has two random Gen objects which produces random Int’s to test the div function.

The random inputs are defined as follows,

val arbitraryInts: Gen[Int] = arbitrary[Int]
val nonZeroInts: Gen[Int]   = arbitrary[Int] suchThat (_ != 0)

arbitrary[Int] is one of the way of producing Gen object which is an abstraction for generating some arbitrary input of given type.

Other few examples are given as below,

Gen.choose(0,100) // Will give you one Int from the range
Gen.oneOf('A', 'E', 'I', 'O', 'U', 'Y') // One from the list
Gen.containerOf[List,Int](Gen.oneOf(1, 3, 5))
// With some condition : any even number from 0 - 200
Gen.choose(0,200) suchThat (_ % 2 == 0)
// Picking any 5 from the range 1 to 6
Gen.pick(5, 1 to 6)
// Even with some distribution of each value
Gen.frequency(
  (3, 'A'),
  (4, 'E'),
  (2, 'I'),
  (3, 'O'),
  (1, 'U'),
  (1, 'Y')
)

All the above ways provides us with Gen object — which will then be used in forAll method to provide the fuzziness to our test cases.

Property Based Testing provides us an easy way to include fuzziness in our testing logic which can’t be evaluated in our traditional way of writing tests. In conventional way we provide set of expected inputs and expected outputs, through which we are not testing the extremes of the method property — this might lead to severe errors in production.

In ScalaCheck we can configure the fuzziness and test the load as well with below properties,

forAll(nonZeroInts, minSuccessful(50)) { n: Int =>
  div(n, n) shouldBe 1
}

forAll method contains variable argument to provide property check configuration to define the test condition. Example above defines that minimum of 50 successful execution to be performed.

maxDiscardedFactor, sizeRange, workers are the other configuration can be used.

ScalaCheck also provides some complex fuzzy creation logics which includes generating the whole case class objects. Refer to the link’s given below for detailed implementation and also integration with ScalaTest.

For Scala developers, ScalaTest and ScalaCheck are pretty good options to write concise and readable test cases real quick.

References:

Example Code

subashprabanantham/hakuna-matata

ScalaTest

ScalaTest

ScalaCheck in ScalaTest

ScalaTest

ScalaCheck User Guide

typelevel/scalacheck