A New Approach to the API End-to-End Testing in Kotlin

Countless times we have implemented integration or end-to-end testing structure for our projects, and the frameworks we use usually provide some structure for it. But almost all of them never felt that we solved what we wanted to solve or a pleasant solution to work with.

@Trendyol, we always check if things are available before implementing any solution; the examples we tried always had crucial problems. For instance, if you search “Spring-Boot Kafka Integration Test/EndToEnd Test” you will get many of the same results requiring peculiar and unuseful plumbing code floating around your code base, which does not solve your problem entirely.

When we started writing Kotlin code, we realized there is no framework/library to test our back-end API end-to-end without having weird plumbing around for all the components we wanted to try — although this applies to almost all languages. At Trendyol, people widely use the Spring-Boot framework with Kotlin, like many, and the ways we found with Spring-Boot e2e testing are not handy. There are lots of annotations that don’t achieve most of the scenarios. Because we usually use multiple physical components to run a code properly when the e2e tests are running. And configuring all of them usually ends up being too ugly.

Other options? Could be docker-compose.yml . But this has drawbacks, too. Hard to set it up, with tangled sh code, execution order problems, challenges for understanding when the environment is ready before starting the tests, lack of assertion over the database, messaging system, lack of mocking external services, and so on.

Whereas it was straightforward for us to address our needs, an e2e testing mechanism:

1. Spins up all the dependencies we have.
2. Is extensible; hence we might add more dependencies in the future
3. Runs the application(Spring, Ktor) physically side-by-side with dependencies, important for CI
4. Provides an assertion mechanism against every component
   - We should be able to check all Kafka Messages, if produced properly, if consumed properly, without exceptions
   - Check all the database values
5. Mocks all external HTTP endpoints and change their behavior according to our e2e test needs
6. Runs on CI
7. Provides the coverage data from the execution
8. Should know as less as possible about the application code and even think of the application as a black-box
9. Easy to write tests
10. Provides a way to keep dependencies running in the local environment after the tests are finalized. So, the next run does not spin up the docker dependencies; hence, it is faster and feels like you are running unit tests!

These items are the birth reasons of the framework. It provides all of them. we wanted to solve all these problems in one place: Stove. — we called it Stove4k in case of a port to another language since we believe that the idea of the Stove is beyond it; we have written it for Kotlin, but the idea itself applies to any language.

How does it work?

The general idea of how Stove works

There is a documentation website you might check later, but the setup code is fairly simple. For example, if we want to spin up the entire system, including our application, the code is below:

 TestSystem(baseUrl = "http://localhost:8001")
     .with {
         httpClient()
         couchbase { CouchbaseSystemOptions("Stove") }
         kafka {
             KafkaSystemOptions(containerOptions = KafkaContainerOptions(tag = "latest"))
         }
         bridge()
         wiremock {
             WireMockSystemOptions(
                 port = 9099,
                 removeStubAfterRequestMatched = true,
                 afterRequest = { e, _, _ ->
                     logger.info(e.request.toString())
                 }
             )
         }
         springBoot(
             runner = { parameters ->
                 stove.spring.example.run(parameters) {
                     this.addTestSystemDependencies()
                 }
             },
             withParameters = listOf(
                 "server.port=8001",
                 "logging.level.root=info",
                 "logging.level.org.springframework.web=info",
                 "spring.profiles.active=default",
                 "kafka.heartbeatInSeconds=2",
                 "kafka.autoCreateTopics=true",
                 "kafka.offset=earliest",
                 "kafka.secureKafka=false"
             )
         )
     }.run()

As you can imagine, withKafka runs Kafka, withCouchbase runs Couchbase, withWireMock runs a WireMock server to mock HTTP requests/responses, and last but not least systemUnderTest is your actual application runner that accepts some CLI arguments like server.port, kafka.heartbeatInSeconds, kafka.offset, logging.level, and so on.

An example test:

test("a test") {
    TestSystem.validate {
        val productCreateRequest = ProductCreateRequest(1L, name = "product name", 99L)
        val supplierPermission = SupplierPermission(productCreateRequest.supplierId, isAllowed = true)
        wiremock {
            mockGet(
                "/suppliers/${productCreateRequest.id}/allowed",
                statusCode = 200,
                responseBody = supplierPermission.some()
            )
        }
        http {
            postAndExpectJson<String>(uri = "/api/product/create", body = productCreateRequest.some()) { actual ->
                actual shouldBe "OK"
            }
        }
        kafka {
            shouldBePublishedOnCondition<ProductCreatedEvent> { actual ->
                actual.id == productCreateRequest.id &&
                    actual.name == productCreateRequest.name &&
                    actual.supplierId == productCreateRequest.supplierId
            }
        }
        couchbase {
            shouldGet<ProductCreateRequest>("product:${productCreateRequest.id}") { actual ->
                actual.id shouldBe productCreateRequest.id
                actual.name shouldBe productCreateRequest.name
                actual.supplierId shouldBe productCreateRequest.supplierId
            }
        }
    }
}

In this test, the paths we tested:

1. An external URL from a different team /suppliers/${productCreateRequest.id}/allowed is mocked with supplierPermission response
2. An actual HTTP request is made against our Spring-Boot Application’s controller. /api/product/create with body and expected string is “OK”
3. As a result of this call, we expect a Kafka message to be published with the type of ProductCreatedEvent with some conditions — this is because there could be multiple tests/messages.
4. Then we expect something in the database.

Here you can see that we tested every aspect of a use case, including all physical edge points.

You can configure Stove according to your application needs; Kafka integration needs a little tweak to see all the messages in e2e testing. All in all, it is a simple integration.

Notes

• All tests can run on CI, and you can collect coverage data.
• This post uses the Kotest testing framework, and shouldBe assertions come from the Kotest framework.
• JDK 16+ is required
• Spring-Boot 2.7.x is required for Spring-Based applications; Stove also supports Ktor.
• There might be configuration and implementation changes in your main codebase to work with Stove. You can think of this as changing a static class implementation into a DI-enabled or a non-static version of the class to make it testable. — Although, this should be very small.

Extensibility

The Stove library is extensible to add more SystemUnderTest to the code base. We have implemented Spring and Ktor applications for now, but you can plug your application in simply by implementing the ApplicationUnderTest interface. You can plug it in if you have already been using Spring Boot.

All the Systems or Components that are used in testing are pluggable. If you have a database the framework didn’t yet implement, you can write it and use it with your testing. The framework will spin up any RunnableSystem. So you are not limited by the framework capabilities and can extend it.

How to get it?

Like every JVM library, you take the necessary libraries. We get the libraries from the Sonar Snapshot repository. Since the framework is getting matured, we constantly publish and update the library, hence the SNAPSHOT versioning strategy. Although it is in use in Trendyol, soon, we will have a stable release in the release repository.

We have also created documentation about how to set it up and get it working.

Take a look at the GitHub repository.

Conclusion

Stove4k will solve your e2e API testing problems and address your needs. It is an evolving library, and it will provide more features in the future. It is already addressing e2e testing needs for our projects and increasing our trust in the code base!

Feedback?

Please share in the comments if you have also struggled with end-to-end testing or have feedback!

Looking forward to hearing your thoughts!

Ready to take your career to the next level?
Join our dynamic team and make a difference at Trendyol.
Want to be a part of our growing company? We’re hiring!
Check out our open positions.