Blackbox API Testing With Kotlin for RESTful Microservices
What does 80% Lines of Code (LOC) Test Coverage get you? How about 100%? They get you some certainty that there was one a piece of code that exercised another piece of code, in the same system, and that most of the system was covered. That is assuming you haven’t configured your Sonar setup to exclude ‘*’.
What’s the problem with that? In part, it’s what we measure, and in part, it’s the way we measure it. In this situation, you simply don’t have any assurance that the things that really matter are being tested. What does it matter that the lug nuts are tight if the car won’t start?
Behavior-Driven Development (BDD) and the notion of “spec” — or specifications — for your tests go some way to alleviate this concern. It changes the focus from how much code is measured, answering instead the question, “Does it do what it is supposed to do?”
When we enter the domain of microservices, we can no longer rely on a self-consistent system, as measured by BDD, as the only measure of functional completeness. Service A and Service B might both be perfect, insofar as their unit tests and BDD tests are concerned, and yet their interoperation from a client perspective may be fundamentally broken. We can make sure the lug nuts are tight and the car starts, but not that the drive-through made our burger correctly.
These days, I will even assert that if I have system-wide coverage through an external blackbox suite of tests (functional integration tests, really), then I have marginalized the value of LOC metrics to a large degree. Of course, I still expect developers to write unit tests and BDD tests — these are just tools that make a service self-consistent and well developed. The real thing that matters to me is that I have Use Case Coverage (UCC) through User Journeys which track interaction with our services in the way an actual client would. These are tests that run the product through its paces end-to-end, across microservices, taking both positive and negative outcomes into account.
There are a few vendors taking a stab at providing this as a tool suite — SmartBear most notably — but this involves licensing their tools and working in a GUI. Of course, we could define these User Journey’s using specifications in Cucumber, but then we have to learn Gherkin, and work within their templated methods.
So, what are our alternatives? Well one is to roll up our sleeves and translate our Acceptance Criteria into User Journeys as code. Let’s make sure that we have the car key, we can get in the car, the wheels are on, the car starts, we can get to the drive-through, order the burger, get home, and validate the burger returns OK 200. (or whatever)
For this to work well, the code should probably live outside of the actual services themselves. Depending on the size and complexity of the application, there may be more than one.
Here Are the Pieces
- Kotlin
The methodology I’ve adopted lately is to use Kotlin with Rest Assured. If you haven’t heard of Kotlin, and you run on JVM, you are missing out. It was created by the folks at JetBrains; and while it was obscure for a bit, with Pivotal starting to really push it traction is imminent. The unfortunately named SparkJava is a microservice framework focused on Kotlin. Anyway — you will hear more and more about Kotlin. It’s like Scala without the sandpaper.
- KotlinTest
Modeled after the fantastic ScalaTest library, KotlinTest smells almost identical; and much like the general language differences, is even easier. It is a Specification-Driven BDD testing kit with plenty of hooks for managing your tests and suites as needed.
- Rest Assured
Rest Assured is a tool to make REST calls and assert their outcomes with minimal friction. Because it is Java, you can use it in Kotlin just fine.
With these three easy pieces, I have everything I need to start building up a fleet of functions that interact with my microservices, and tests that create User Journey’s based on those functions.
Tutorial
If you just want to dig into the code, start here: https://github.com/revelfire/blackboxmicroservicetesting
Note — Specific instructions for running this code are in the readme.md of that repo and will not be repeated here.
Stack
· Gradle
· Kotlin
· KotlinTest
· Intellij Kotlin Plugin
· RestAssured
That’s really about it.
Structure
We break the code up into folders that match structure-specific functionality.
/functions
This folder is specifically for Functions that act as clients for the API. There should generally not be tests here, just Functions.
/tests
This folder is specifically for single Tests/scenarios that validate a particular API Function invocation. These will include positive and negative test cases.
/journeys
This folder is specifically for tests that composite multiple Functions into User Journeys that should ideally mirror application use cases that have documented acceptance criteria.
Pieces
build.gradle
This file stitches all the pieces together and gives us the simple text execution.
buildscript {
ext.kotlin_version = "1.1.4"
repositories {
mavenLocal()
mavenCentral()
}
dependencies {
classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
}
}repositories {
mavenLocal()
mavenCentral()
}apply plugin: 'java'
apply plugin: 'idea'
apply plugin: 'maven'
apply plugin: 'kotlin'group 'com.revelfire.test.api'
version '1.0-SNAPSHOT'sourceCompatibility = 1.8
targetCompatibility = 1.8dependencies {
compile group: 'org.jetbrains.kotlin', name: 'kotlin-stdlib', version: "$kotlin_version"//Include our app lib so we have access to models (not mandatory, but helpful)
compile group: 'com.revelfire', name: 'car-service', version: '1.0-SNAPSHOT'compile group: 'com.fasterxml.jackson.module', name: 'jackson-module-kotlin', version: '2.9.0'compile group: 'io.rest-assured', name: 'rest-assured', version: '3.0.2'
compile group: 'org.hamcrest', name: 'hamcrest-core', version: '1.3'
compile group: 'org.jetbrains.kotlin', name: 'kotlin-stdlib', version: "$kotlin_version"
compile group: 'io.kotlintest', name: 'kotlintest', version: "$kotlin_version"
compile group: 'com.github.javafaker', name: 'javafaker', version: "0.13"
testCompile group: 'io.kotlintest', name: 'kotlintest', version: '2.0.1'
}tasks.test.doFirst {
def includeTags = System.properties['includeTags']
if(includeTags) {
systemProperty 'includeTags', includeTags
}
testLogging.events = ['passed','failed','skipped']
}tasks.test.outputs.upToDateWhen { false }
TestBase.kt
This file has some basic underpinnings for test execution and hooks for overarching functionality.
package app//…imports omitted…object QuickRun : Tag()
object Integration : Tag() //Represents a single unit of testing an API endpoint (various scenarios possibly)
object Journey : Tag() //Represents a group of tests that mimic a user behavior/journey through the appobject TestBase {var props:Properties = Properties();init {
props = readProperties()
validateProperties()
}fun carServiceHost():String = "${getProperty("service.car-service.url")}:${getProperty("service.car-service.port")}"
fun foodServiceHost():String = "${getProperty("service.food-service.url")}:${getProperty("service.food-service.port")}"fun validateProperties() {}fun getPropertyAsInt(key:String):Int {
val value = getProperty(key)
value?.let {
return value.toInt()
}
throw RuntimeException("Property ${key} not found!")
}fun getPropertyAsBoolean(key:String):Boolean {
val value = getProperty(key)
value?.let {
return value.toBoolean()
}
throw RuntimeException("Property ${key} not found!")
}fun getProperty(key:String):String? {
System.getProperty(key)?.let {
return System.getProperty(key)
}
return props[key] as String
}fun getRequiredProperty(key:String):String? {
System.getProperty(key)?.let {
return System.getProperty(key)
}
props[key]?.let {
return props[key] as String
}
throw RuntimeException("Property ${key} marked as required but not present!")
}fun readProperties():Properties = Properties().apply {
FileInputStream("src/test/resources/testing.conf").use { fis ->
load(fis)
}
}}//Kotlintest interceptors that fire before the entire suite, and after.
object GlobalTestSuiteInitializer : ProjectConfig() {private var started: Long = 0override fun beforeAll() {
RestAssured.config = RestAssuredConfig.config().objectMapperConfig(ObjectMapperConfig.objectMapperConfig().jackson2ObjectMapperFactory(objectMapperFactory))
started = System.currentTimeMillis()
}override fun afterAll() {
val time = System.currentTimeMillis() - started
println("overall time [ms]: " + time)
}
}
Note in particular the GlobalTestSuiteInitializer above which can be used for any sort of suite-wide bootstrapping.
RestAssured Configuration (RestAssuredSupport.kt)
We have a couple of twiddly bits we need to set up for JSON manipulation and other things.
package app.util//…imports omitted…interface RestAssuredSupport {
fun RequestSpecification.When(): RequestSpecification {
return this.`when`()
}
fun ResponseSpecification.When(): RequestSender {
return this.`when`()
}}//http://www.programcreek.com/java-api-examples/index.php?api=com.jayway.restassured.config.RestAssuredConfigobject ObjectMapperConfigurator {
val objectMapper = ObjectMapper().registerModule(KotlinModule())
init {
objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
objectMapper.configure(DeserializationFeature.FAIL_ON_MISSING_CREATOR_PROPERTIES, false)objectMapper.setSerializationInclusion(JsonInclude.Include.NON_EMPTY)
}
fun get():ObjectMapper {
return objectMapper
}
}val objectMapperFactory = object : Jackson2ObjectMapperFactory {override fun create(cls: Class<*>?, charset: String?): ObjectMapper {
return ObjectMapperConfigurator.get()
}
}
Finally
gradle test
Done.
As stated above — the real thing that matters to us is that we have Use Case Coverage (UCC) through User Journeys and tests that run the product through its paces end-to-end, across microservices. This method of testing RESTful microservices with Kotlin will help get us there.
DISCLOSURE STATEMENT: These opinions are those of the author. Unless noted otherwise in this post, Capital One is not affiliated with, nor is it endorsed by, any of the companies mentioned. All trademarks and other intellectual property used or displayed are the ownership of their respective owners. This article is © 2017 Capital One.
