Enhance Component Test with API Coverage using Karate
What is Karate ?
Karate is the only open-source tool to combine API test-automation, mocks, performance-testing and even UI automation into a single, unified framework. The syntax is language-neutral, and easy for even non-programmers. Assertions and HTML reports are built-in, and you can run tests in parallel for speed.
In our opinion, Karate is a GREAT Test Automation solution that helps development teams achieve the “Shift-Left” objective and provide trust on the products they build.
This article is based on the experience of developing Java Spring Boot Microservices in a complex environment (around 50 different development teams building a distributed platform for a Tier 1 Financial Institution)
Why Karate ?
There are plenty of different Test Automation frameworks out there and sometimes is difficult to decide which one better fits your requirements.
The recommendation here would be to spend some time reviewing the documentation, check their capabilities, try them using a simple use case and do a comparison of your results.
In our case, there were some key features that Karate offered that convinced us to choose it instead of others. Main ones:
- Easy to code and easy to read
- All-in-one. Mocking included. We didn´t want to use another solutions for mocking
- Parallel test executions supported
- Extensible to support non-standard use cases
- Reporting capabilities. Basic for execution and extensible to create functional reports
Reviewing Karate documentation we found most of the requirements fulfilled. Among them there is one specifically, which is Java Interoperability, that opens the door for any extension you could imagine.
Before creating API Coverage Report…
We did an exhaustive review of all the features offered out of the box by Karate and completed the following milestones (which are not part of this article) before starting to develop the API Coverage Report. Listed here:
- Developed a library on top of Karate extending out-of-the-box functionalities. For example, connectivity from Karate features to Kafka producer/consumer, Databases, AWS S3 buckets, Redis cache, SQS clients, etc…
- Embedded surrounding components (as mentioned in previous point) as part of the Microservice context to isolate from real / external ones
- Used Karate Mock Servers extensively to isolate from any other external service
- Executed Spring Boot Microservices locally without any external dependency and execute Karate scenarios (feature files) against the service (through exposed APIs)
- Generated Functional Coverage Report based on expected scenarios agreed with Product Owners to provide transparency (see an example)
All of the mentioned items contributed to achieve a real “Shift-Left” approach and transformed Component Test phase (run in developer´s local machines) in a very powerful stage.
Include API Coverage indicator
We realized there was another dimension missing in this dashboard, which is the actual API endpoints exposed by the service under test and covered by our Component Tests. This is obviously another key indicator of the actual functionality covered by our test scenarios.
Different ways to achieve that but, in our case, we decided to use the Runtime Hook functionality (link to class) which is not clearly documented in the official Karate github site but a really interesting feature.
This allowed us to define actions for each of the available execution hooks (ie. beforeSuite, beforeFeature, beforeScenario, beforeStep, and the corresponding after hooks).
Let me put in a sequence the steps we followed:
1 — Capture ALL the REST calls made from the scenarios to the service
Override the afterStep (StepResult result, ScenarioRuntime sr) method, that contains ALL the details we need to intercept REST calls. This is done by filtering SteResults that use “method” keyword and using the ScenarioRuntime to get the actual REST request details.
The next step is straightforward, i.e. store the REST calls (in memory or a file) so that at the end of the test execution, we have ALL the calls identified and documented.
2 — Identify ALL The REST endpoints exposed by the service
At this point, it is assumed that the API specification file for the service (swagger in this case) is available for our process (in the classpath, for example). This file will be used to get the list of available endpoints.
3 — Merge the information
With all the ingredients in place (service API specification and the actual REST calls) we can use the afterSuite hook to generate the data that will be used by the dashboard. This data could contain:
Find just a log with the bare minimum information that we could be providing.
4 — Enhance Functional Coverage Report
As the last step, we could include this information in the Functional Coverage Report to provide that Extra Point of View.
Conclusion
In this post, we described how to integrate API Coverage indicator in a Component Test Report, adding this extra layer of quality to the products we are building. We also explained the reasoning behind choosing Karate as the tool to help us achieve this objective.
Keep Testing !!
