Suppressing KotlinTest mutiny

Paweł Weselak
Mar 14 · 8 min read

Frequent KotlinTest issues and how to fix them

Pirate skull inviting only true pirates
Pirate skull inviting only true pirates
Photo by Mateusz Dach on Pexels

Intro

Our team currently started cruise to develop the next version of ‘the API’ and a new set of services emerged during that journey. We have a free hand to sail to the open waters of technologies ocean, so we can choose the best fit for our project. Our ship wharfed to an island named Kotlin language. Very quickly we discovered KotlinTest library on this land. When fighting the battle of writing test cases to our production code, we had a mutiny incited by KotlinTest. In this article, I want to describe the course of the rebellion and how our brave team suppressed it.

In the release 4.0.0, the library changed the name from KotlinTest to Kotest. As of writing the article a stable version of Kotest hasn’t been released yet.

Why KotlinTest at all?

In terms of writing tests in Kotlin you have several test libraries to choose. The most common are Spek, JUnit5 (preferably with some assertion library like AssertJ or Hamkrest), and of course KotlinTest. As Dear Reader probably already anticipated, in our project we decided to use KotlinTest. The authors of the library are inspired by the test library for Scala — ScalaTest. Our team promotes functional programming, so do KotlinTest team. The library has many useful features like different styles of writing test specifications, soft assertions or data-driven testing. It is also quite popular in the community, actively developed and well supported.

Minor but persistent

There are a few annoying things about KotlinTest. First of all, integration with IntelliJ — the IDE which we use on a daily basis, is not so good. It is impossible to execute a test case of one’s choice from test specification. I tried the official IntelliJ plugin with no avail. Thanks to my project manager I realized that the plugin to run things smoothly requires some additional configuration. And it wasn’t straightforward for me. To set it up, first install a plugin named KotlinTest. After that, if you are using Gradle in IntelliJ preferences under select . Then upon running a test, you can choose to run it with the KotlinTest plugin. If it doesn't work try to use Restart/Invalidate cache option.

Before the moment I started using KotlinTest plugin, I had found in the documentation mention about prefixing test case with “f:”. It should do the trick as it is done in popular JavaScript libraries. Prefix “f:” means focused here. Unfortunately, this tip doesn’t work well. Neither standard IntelliJ runner nor Gradle will execute any tests if I try to focus any test case. IntelliJ informs that there are no tests received.

import io.kotlintest.shouldBe
import io.kotlintest.specs.StringSpec
class FocusedStringSpec : StringSpec({

"f:I want to focus this method" {
2 + 2 shouldBe 4
}
"test method" {
2 + 2 shouldBe 4
}
})

Surprisingly bang tests with exclamation mark “!” works. It excludes the test case from running. So if want to focus some test case I just exclude all the others. 😜

import io.kotlintest.shouldBe
import io.kotlintest.specs.StringSpec
class BangStringSpec : StringSpec({

"!Bang this method" {
2 + 2 shouldBe 4
}
"test method"() {
2 + 2 shouldBe 4
}
})

Again this feature works perfectly fine with the KotlinTest plugin. This can be solved also by adding KotlinTest Gradle plugin to your project. Check out the discussion in the GitHub issue.

Duplicated test name

There are more issues that lead to a situation where tests should be executed, but they don’t. It is a serious problem when a test certainly should fail. We observed that behavior when the name of the test case method was duplicated by mistake. Any test case of the specification sadly didn’t run. It caused the whole build was green, but actually it shouldn’t. The problem was masked and not easy to spot.

import io.kotlintest.shouldBe
import io.kotlintest.specs.StringSpec
class DuplicatedStringSpec : StringSpec({ "duplicated method" {
2 + 2 shouldBe 4
}
"duplicated method" {
2 + 2 * 2 shouldBe 6
}
})

The root cause of the issue was the tiny dependency hell we had in the project. 😉 To be more precise, Kotlin depends on arrow-core-data in version 0.9.0 while our project uses arrow 0.10.x. It leads to the replacement of the arrow-core-data version on which KotlinTest is dependent.

dependencyManagement {
dependencies {
dependencySet("io.arrow-kt:0.10.3") {
entry("arrow-core-data")
}
dependencySet("io.kotlintest:3.4.2") {
entry("kotlintest-runner-junit5")
entry("kotlintest-extensions-spring")
}
}
}

When Spring eventually comes…

A similar issue we could encounter while running integration tests with Spring. When the creation of the Spring context is failed then none of the test cases is executed from the test. This leads to falsely green build and is hard to notice. The cause of the issue was the same as in the “Duplicated test name” section.

One digression on testing with Spring. We don’t like much autowiring variables approach. Mutable state is not something we want to have in our functional code even if it is in the tests. Hopefully, KotlinTest allows us to wire through the constructor of test spec. To enable that, Spring project extension needs to be added to the project. This approach has some drawbacks although. The project listeners/extensions are executed before test listeners. So if you have any test setup that Spring context creation depends on, you cannot put it in the test listeners. In that case, we use Spring test context configuration with post construct and post destroy lifecycle methods. In the last resort, one can go back to vars.

Strange behavior

Talking about test listeners there is one thing worth mentioning. If you take advantage of BehaviorSpec, most likely you want to run test listener methods before and after whole given/when/then test case. Use method, otherwise and methods will execute not only on , but also on and blocks.

import io.kotlintest.TestCase
import io.kotlintest.shouldBe
import io.kotlintest.specs.BehaviorSpec
class IsTopLevelSpec : BehaviorSpec({ Given("2 and 2") {
When("sum") {
val result = 2 + 2
Then("result is 4") {
result shouldBe 4
}
}
}
Given("2, 2 and 2") {
When("2 + 2 * 2") {
val result = 2 + 2 * 2
Then("result is 6") {
result shouldBe 6
}
}
}
}) {
override fun beforeTest(testCase: TestCase) {
if (testCase.isTopLevel()) {
println("It will be printed before entering top-level"
+ testCase.name)
}
println("It will be printed before entering"
+ testCase.name)
}
}

Another strange thing is the way how test results are reported in IntelliJ after running Gradle test task. Only “Then” names are listed on the report. So if you have several “Thens” labeled the same way in your Test specification, it is not easy to distinguish them. The problem can be solved with KotlinTest plugin. Running the tests using this plugin will show the nested structure of given/when/then on the report.

One can complain about the tree-like structure of BehaviorSpec. As a matter of fact, it is a definite advantage of the library. Thanks to that Kotlin compiler itself can check the right order of clauses. In JUnit you can put labels or comments, but they don’t have any meaning. While in Spock, a test library for Groovy, checking of labels order is added by the library.

Are you still hiding something from me, KotlinTest?

Please be careful when you are going to use AssertSoftly. Normally all assertions in the assert softly block will be executed even if the first one fails. One can meet yet another concealed troubles there. Assert softly works well with KotlinTest matchers. But if you want to combine other testing tools like reactor-test for example, they most likely throw some kind of AssertionException when assertion fails. It will cause that assert softly block will be interrupted. If you don’t always follow TDD completely you may not notice this “feature” in the first place.

import io.kotlintest.*
import io.kotlintest.specs.StringSpec
import org.assertj.core.api.Assertions.assertThat
class AssertSoftlySpec : StringSpec({
"should execute all assertions, but it doesn't" {
assertSoftly {
assertThat("You will never know...").startsWith("I")
2 + 2 shouldBe 5
}
}
})

To avoid that, you probably want to implement your custom matchers. In your matcher you can catch AssertionException in the overridden test method. Then return boolean indicating success or failure of the assertion.

import io.kotlintest.*
import io.kotlintest.specs.StringSpec
import org.assertj.core.api.Assertions
class AssertSoftlyWithCustomMatcherSpec : StringSpec({ "will execute all assertions" {
assertSoftly {
"You will never know..." should startsWith("I")
2 + 2 shouldBe 5
}
}
})
private fun startsWith(prefix: String) = object : Matcher<String> {
override fun test(value: String): MatcherResult =
try {
Assertions.assertThat(value).startsWith(prefix)
matcherResult(passed = true, prefix = prefix)
} catch (e: AssertionError) {
matcherResult(passed = false, prefix = prefix)
}
}
private fun matcherResult(passed: Boolean, prefix: String) =
MatcherResult(
passed,
"doesn't starts with $prefix",
"shouldn't start with $prefix"
)

Another annoying thing is related to method usage. In behavioral tests, it is defined for but not for method. I like starting test level names with capital letter since is a reserved keyword in Kotlin. In order to compile your code, you need to put it in backticks:. So instead of , , one can use , , . Except you want to add config to the test. In this case, you have to use , , which looks silly.

import io.kotlintest.shouldBe
import io.kotlintest.specs.BehaviorSpec
class ConfigOnThenSpec : BehaviorSpec({ Given("Given") {
When("When") {
then("then must start with lowercase")
.config(invocations = 5) {
2 + 2 shouldBe 4
}
}
}
given("given") {
`when`("when in backticks in order to compile") {
then("then").config(invocations = 5) {
2 + 2 shouldBe 4
}
}
}
})

Summary

Is our journey with KotlinTest over? I hope not. Probably we will discover even more issues, but despite its peculiarities, KotlinTest is a promising library. The development is very active and the releases are coming quite frequently. We optimistically see the future of the library. In the 4.0.x release, the library is changing the name from KotlinTest to Kotest. It is already available. In the next article, I will describe the migration steps needed. And also I will check if the above issues have been fixed.

How about you? I strongly encourage you, dear reader, to check the library on your own. Maybe you have some other solutions to share? Feel free to post your comment below the article.

A complete project with code samples is available on my GitHub.

Sailing ship reaching the safe bay
Sailing ship reaching the safe bay
Photo by Pixabay on Pexels

Originally published at https://pawelweselak.com on March 14, 2020.

VirtusLab

Virtus Lab company blog

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store