How to improve the quality of tests using property-based testing
In the last article I showed how mutation testing can be used to check the tests, that are written for an application, for quality regarding detection of changes in productive code. Mutation testing is a testing type, which does not run against the productive code to check for the correct behavior, but changing the productive code to check if tests recognize the change.
Today I want to introduce another testing type, which can help to increase the quality of the tests, but this time directly inside the existing tests by adding randomness — Property-based testing.
Introduction
What is Property-based testing? Property-based tests are designed to test the aspects of a property that should always be true. They allow for a range of inputs to be programmed and tested within a single test, rather than having to write a different test for every value that you want to test. Property-based testing is a form of fuzzing (Fuzz testing). The main goal is to provide random input data to tests, so that all possible boundaries are covered. Instead of testing only for by developer specified input data, all as valid defined input data is used.
This is a very theoretical description, to better understand this, let’s look on an example. The focus is on showing how Property-based testing is working, not showing a real world example:
val minDate: LocalDate = LocalDate.of(2022, 1, 1)
val maxDate: LocalDate = LocalDate.of(2022, 12, 31)
class ApplicationService {
fun createOutput(inputDto: InputDto): OutputDto {
return mapInputToOutput(inputDto)
}
private fun mapInputToOutput(inputDto: InputDto): OutputDto {
return OutputDto(
date = inputDto.date,
amount = inputDto.amount,
positions = mapInputPositionToOutputPosition(
positions = inputDto.positions
)
)
}
private fun mapInputPositionToOutputPosition(positions: List<InputPositionDto>): List<OutputPositionDto> {
require(positions.isNotEmpty()) {
"Positions must not be empty."
}
val sum = positions.fold(0.0) { acc, next -> acc + next.value }
require(sum.compareTo(0.0) > -1) {
"Sum of positions amount must be greater than 0.0 but is $sum"
}
return positions.map {
OutputPositionDto(
name = it.name,
value = it.value.toEURCurrency()
)
}
}
}
data class InputDto(
val date: LocalDate,
val amount: Int,
val positions: List<InputPositionDto>
)
data class InputPositionDto(
val name: String,
val value: Double
)
data class OutputDto(
val date: LocalDate,
val amount: Int,
val positions: List<OutputPositionDto>
) {
init {
require(date in minDate..maxDate) {
"Date '${date}' must be within '$minDate' and '${maxDate}'."
}
require(amount >= 0) {
"Amount '$amount' must be greater or equal null."
}
}
}
data class OutputPositionDto(
val name: String,
val value: MonetaryAmount
)
fun Double.toEURCurrency(): MonetaryAmount {
return BigDecimal(this, MathContext(2, RoundingMode.HALF_UP)).ofCurrency<FastMoney>("EUR".asCurrency(), typedMonetaryContext<FastMoney> {
setPrecision(2)
})
}The code is showing a kind of simplified ApplicationService which is mapping a given input DTO object to an validated output DTO object. For the different properties and data of the input DTO there are validations available, which will lead to throwing an exception during mapping process, if they are not all fulfilled.
The classical way for testing this functionality by Unit Tests is to write for every potential branch in the code a separate test case. This will look something similar to below version:
internal class ApplicationServiceTest {
private val applicationService = ApplicationService()
@Test
fun `throws exception if date is below minDate`() {
// given
val inputDto = createValidInputDto().copy(
date = LocalDate.of(2020, 1, 1)
)
// when + then
val exception = shouldThrow<IllegalArgumentException> {
applicationService.createOutput(inputDto)
}
exception.message shouldBe "Date '2020-01-01' must be within '2022-01-01' and '2022-12-31'."
}
@Test
fun `throws exception if date is above maxDate`() {
// given
val inputDto = createValidInputDto().copy(
date = LocalDate.of(2025, 1, 1),
)
// when + then
val exception = shouldThrow<IllegalArgumentException> {
applicationService.createOutput(inputDto)
}
exception.message shouldBe "Date '2025-01-01' must be within '2022-01-01' and '2022-12-31'."
}
@Test
fun `throws exception if amount is below zero`() {
// given
val inputDto = createValidInputDto().copy(
amount = -1
)
// when + then
val exception = shouldThrow<IllegalArgumentException> {
applicationService.createOutput(inputDto)
}
exception.message shouldBe "Amount '-1' must be greater or equal null."
}
@Test
fun `throws exception if sum of positions amount is below zero`() {
// given
val inputDto = createValidInputDto().copy(
positions = listOf(
InputPositionDto(
name = "First Position",
value = 2.3
),
InputPositionDto(
name = "Second Position",
value = -12.2
)
)
)
// when + then
val exception = shouldThrow<IllegalArgumentException> {
applicationService.createOutput(inputDto)
}
exception.message shouldBe "Sum of positions amount must be
greater than 0.0 but is -9.899999999999999"
}
@Test
fun `returns outputDto for valid input`() {
// given
val inputDto = createValidInputDto()
// when
val actual = applicationService.createOutput(inputDto)
// then
actual shouldBe OutputDto(
date = LocalDate.of(2022, 1, 1),
amount = 2,
positions = listOf(
OutputPositionDto(
name = "First Position",
value = 2.3.toEURCurrency()
),
OutputPositionDto(
name = "Second Position",
value = 12.2.toEURCurrency()
)
)
)
}
}
private fun createValidInputDto() = InputDto(
date = LocalDate.of(2022, 1, 1),
amount = 2,
positions = listOf(
InputPositionDto(
name = "First Position",
value = 2.3
),
InputPositionDto(
name = "Second Position",
value = 12.2
)
)
)Even in this easy example it is necessary to write quit a lot of tests to cover all potential cases (e.g. calculation of sum of a lot of positions with different precision). With increasing complexity it gets difficult to cover all potential combinations in tests.
So to improve the tests and reduce the amount of tests in total, it is possible to use functionality, which is provided by the Junit5 test framework out of the box, and write parameterized tests.
class ApplicationServiceParameterizedTest {
private val applicationService = ApplicationService()
@ParameterizedTest
@MethodSource("createInputData")
fun `returns outputDto for valid input`(inputDto: InputDto, outputDto: OutputDto) {
// when
val actual = applicationService.createOutput(inputDto)
// then
actual shouldBe outputDto
}
companion object {
@JvmStatic
private fun createInputData() = Stream.of(
Arguments.arguments(
InputDto(
date = LocalDate.of(2022, 1, 1),
amount = 2,
positions = listOf(
InputPositionDto(
name = "First Position",
value = 2.3
),
InputPositionDto(
name = "Second Position",
value = 12.2
)
)
),
OutputDto(
date = LocalDate.of(2022, 1, 1),
amount = 2,
positions = listOf(
OutputPositionDto(
name = "First Position",
value = 2.3.toEURCurrency()
),
OutputPositionDto(
name = "Second Position",
12.2.toEURCurrency()
)
)
)
),
Arguments.arguments(
InputDto(
date = LocalDate.of(2022, 12, 31),
amount = 8,
positions = listOf(
InputPositionDto(
name = "First Position",
value = -0.0003
),
InputPositionDto(
name = "Second Position",
value = 0.0004
)
)
),
OutputDto(
date = LocalDate.of(2022, 12, 31),
amount = 8,
positions = listOf(
OutputPositionDto(
name = "First Position",
value = (-0.0003).toEURCurrency()
),
OutputPositionDto(
name = "Second Position",
(0.0004).toEURCurrency()
),
)
)
),
Arguments.arguments(
InputDto(
date = LocalDate.of(2022, 1, 1),
amount = 2,
positions = listOf(
InputPositionDto(
name = "First Position",
value = -122.30
),
InputPositionDto(
name = "Second Position",
value = 122.31
)
)
),
OutputDto(
date = LocalDate.of(2022, 1, 1),
amount = 2,
positions = listOf(
OutputPositionDto(
name = "First Position",
value = (-122.30).toEURCurrency()
),
OutputPositionDto(
name = "Second Position",
122.31.toEURCurrency()
)
)
)
)
)
}
}The problem with this kind of parameterized tests, the developer has to specify the input and the output which should be entered to the test method and validated the result against. So the coverage of all potential combinations of input values still depends on the developer. The question is: Can a developer know all potential input combinations? In the sample code this is possible but productive code is in most cases a lot more complex. So instead of spending time to define all potential test cases manually, let’s have a look how Property-based testing can help to solve this problem.
Property-based Testing
Property-based testing focus on common properties of the code under test. This properties can be:
- result is always an exception
- result is always inside a special range
- result does always successfully store data to database
- …
The concrete result (e.g. a special amount) cannot be asserted in this kind of tests because the input is randomly generated. I’m not interested in this concrete values but in one of the above conditions.
To make this more pratical, let’s see how this can look in the original example of above.
Kotest
For using Property-based testing in Kotlin application there are mainly 2 common frameworks available:
In this article I use Kotest for the implementation of Property-based tests mainly because I also use the framework for assertion in the tests and its a Kotlin native framework. I always try to focus on Kotlin native framework, if they are available and offer the same possibility as the classical Java counterpart.
For using the functionality in the application I need to add a dependency to the build.gradle.kts configuration file.
testImplementation("io.kotest:kotest-property:5.5.4")After importing the dependency in the project everything is ready to use the Kotest module.
Let’s start with a basic example to understand the functionality of the tests. To have a similar setup of the tests as Junit5, I use the AnnotationSpec style for the tests. Kotest provides different styles of tests, which can be used depending on personal preference. An overview about the supported styles can be found in documentation.
There are 2 possible variants possible for the Property-based tests:
- checkAll
The test will pass if for all given input values true is returned - forAll
The test wil pass if for all given input values no exception is thrown.
An example of both variants can be found below:
class MyTestClass : AnnotationSpec() {
@Test
suspend fun checkStringLength() {
forAll<String, String> { a, b ->
//...
(a + b).length == a.length + b.length
}
}
@Test
suspend fun checkDivideOperation() {
checkAll<Int, Int>{a, b ->
//...
(a / b)
}
}
}For both you specify the amount of input values with type and inside of the function body the concrete test happens. In the forAll example there is an check which should result in true as last statement and in the checkAll example it’s only checked that no exception is thrown during execution of the statements.
In above example Kotest is providing random input values for the String and Int type out of the box.
By default there are 1000 iterations done per test case. This value can be specified either globally
class MyTestClass : AnnotationSpec() {
init {
PropertyTesting.defaultIterationCount = 500
}
@Test
suspend fun checkStringLength() {
forAll<String, String> { a, b ->
(a + b).length == a.length + b.length
}
}
@Test
suspend fun checkDivideOperation() {
checkAll<Int, Int>{a, b ->
(a / b)
}
}
}or by configure it per test case
class MyTestClass : AnnotationSpec() {
@Test
suspend fun checkStringLength() {
forAll<String, String>(500) { a, b ->
(a + b).length == a.length + b.length
}
}
@Test
suspend fun checkDivideOperation() {
checkAll<Int, Int>(700){a, b ->
(a / b)
}
}
}As expected the second test case fails if 0 is used for b variable.
Now that the test fails, because one combination of input data is leading to an exception, I need the possibility to get the concrete input. For this simple case with primitive Int as input it is easy to find the information in the log output.
But in case I have a more complicated input data (like nested objects) it can help to re-run the test with only the input which leads to failing test (e.g. to debug). For this the failing test is printing a so called “seed”. This seed can be entered to configuration of test. With this I can reduce the test to one failing case and be able to debug for the source.
@Test
suspend fun checkDivideOperation() {
checkAll<Int, Int>(PropTestConfig(seed = 2089844800)){ a, b ->
(a / b)
}
}Until now I just showed a very basic example to introduce the basics of property-based testing with Kotest. Now let’s go back to the initial example.
In this case there is no primary data type required as input data but an DTO object with a nested structure. How to deal with this in Kotest?
Kotest provides a way to compose complex data structures by so called “Arb” generators of build-in types.
A first version looks like below:
class ApplicationServicePropertyBasedTest : AnnotationSpec() {
private val applicationService = ApplicationService()
@Test
suspend fun `createInput not throws exception for valid input`() {
checkAll(createInputArb()) { inputDto ->
applicationService.createOutput(inputDto)
}
}
}
fun createInputArb() = arbitrary {
InputDto(
date = dateArb.bind(),
amount = amountArb.bind(),
positions = createInputPositionsArb().bind()
)
}
fun createInputPositionsArb() = arbitrary {
val positions = mutableListOf<InputPositionDto>()
repeat(amountPositionsArb.bind()) {
positions.add(createInputPositionArb().bind())
}
positions.toList()
}
fun createInputPositionArb() = arbitrary {
InputPositionDto(
name = nameArb.bind(),
value = valueArb.bind()
)
}
private val dateArb = Arb.localDate(minDate = minDate, maxDate = maxDate)
private val amountArb = Arb.int(min = 0, max = Int.MAX_VALUE)
private val amountPositionsArb = Arb.int(min = 1, max = 100)
private val valueArb = Arb.double()
private val nameArb = Arb.string(minSize = 1, maxSize = 100)Instead of using the syntax for the build-in data types for the checkAll function I provide an Arb of type InputDto. This is providing a randomly generated input for the service. The input is composed of build-in random data types like Double, Int or LocalDate, which can be customized by e.g. giving a minimum or maxium value or more complex rules.
But when running the test I get an exception because the generators are not configured correctly.
In the above case I get an ArithmeticException because of an overflow. So I need to update the Arb generator for the value of the InputPositionDto to limit its input.
private val valueArb = Arb.double(min = -10000.0, max = 10000.00)After update and re-running the test, I still get an exception because the generated InputPositionDto sums are not valid according to the requirement.
For fixing this issue I need to add a condition for the creation of the InputPositionDto objects to fullfil the general requirement. I need to do some changes to the logic for the creation of random amount of InputPositionDto in order to have always a positive total amount.
fun createInputPositionsArb() = arbitrary {
val positions = mutableListOf<InputPositionDto>()
val amountOfPositions = amountPositionsArb.bind()
var totalSum = 0.0
repeat (amountOfPositions) {
val value = valueArb.bind().setScale(2, RoundingMode.HALF_UP).toDouble()
totalSum += value
positions.add(createInputPositionArb(value).bind())
}
if (totalSum < 0.0) {
positions.add(createInputPositionArb(abs(totalSum)).bind())
}
positions.toList()
}This function is an example how I can include logic for creation of the random input to fulfill the business logic.
With this change the tests succeed as expected. With this tests I can run 1000 random test cases against the service in just about 1.5 seconds.
This article is just an introduction to property-based testing using Kotest. Kotest is providing a lot of additional advanced functionality, which can help to adapt the property-based tests to your own needs, like Assumptions, Statistics or generators for Arrow. You can find a good explanation in the official documentation of Kotest
The code I used for the article can be found on Github: https://github.com/PoisonedYouth/kotlin-propertybased-testing
