Different levels of API contract testing with Microcks
As a follow-up to my previous Microcks and Pact for API contract testing post, I’m still delving into the mysteries of what API contract testing means. Even if I give the pride of place to schema conformance testing in the first post, I think the picture has to be completed with other capabilities we have in Microcks.
Today’s complement is based on a truth I voluntarily left aside previously: even if they may represent a large part, schemas cannot be considered full-blown contracts on their own. Schemas are only a portion of what a service provider has to abide by to allow consumers to fully exploit the functionality it offers. Lots of rules cannot be translated into schema whatever the technology.
“Contracts” are then just more than schemas that define the interactions between a provider and consumers. They must also allow us to check the expectations of how a provider behaves. In the case of an automated system, this can especially be done by inspecting response data when introducing variation in request.
That said, contract testing should allow checking two main kinds of conformance:
- Syntactic conformance: the provider actually respects the structure of messages it exposes to consumers. This means that consumers will be technically able to interact with the API or service,
- Behavior conformance: the provider actually provides the data that is expected to behave consistently regarding the business domain. This means that consumers will be able to use the API or service to built-in their own service and create new business value.
In this blog post, we are going to illustrate how Microcks can be used to ensure both kinds of conformance. For that, we’ll use two different kinds of artifacts to contract-test an API implementation:
- an OpenAPI Schema to perform the syntactic conformance check,
- a Postman Collection to perform the behavior conformance validation.
Through a simple use-case and scenario, we’ll demonstrate that a schema only cannot be used for validating the expected rules of an API application. Let’s start!
Use-case and scenario
Our use-case is fairly simple: it’s the mandatory API everyone must have in its catalog: an API about pastries 🍰 🥧 🧁
A Pastry is a pretty basic object: it has a name, a description, a size (t-shirt one 😉), a price, and a status telling us if available or out of stock. Our API will hold three basic operations that allow us to list pastries using a mandatory size parameter, get the details on a pastry using its name or update this pastry price (who said “inflation time”?).
We will then deploy and test three different versions of the same application implementing this API. Each version represents a different level of conformance and will then illustrate the issues we’ll be able to detect with different kinds of contract tests.
For the sake of simplicity, we provide the three different versions of the application through container images that you may run locally using Docker Desktop, Podman, or some other alternative. The source code of the different versions is also available in this GitHub repository. We also assume you’re running Microcks using the same method as explained on the Getting Started page.
Testing Version 01
First thing first, let’s register the API Pastry into Microcks. To do that, you’ll have to import this OpenAPI Schema we created to describe the API syntactic contract. To import if either realize a direct upload or create a scheduled import in Microcks. You’ll get the following result in the Microcks UI:
Some things of importance to notice here:
- Mock endpoints are immediately available at http://localhost:8080/rest/API+Pastries/0.0.1/ (you can try out http://localhost:8080/rest/API+Pastries/0.0.1/pastries?size=L if you want)
- There’s one OpenAPI contract attached to the API,
- The conformance score is
0.0%(we didn’t run any tests so far).
In order to launch a first conformance test, you’ll have to deploy Version 01 of the application. As explained in the use-case introduction, you need to run the container with label 01, mapped on local port 3001:
$ docker run -it -p 3001:3001 quay.io/microcks/contract-testing-demo:01
=== OUTPUT ===
Unable to find image 'quay.io/microcks/contract-testing-demo:01' locally
01: Pulling from microcks/contract-testing-demo
b964235f9f30: Already exists
947a9cb9218d: Already exists
1128b5439c0c: Pull complete
d0ac8fc3cde2: Pull complete
ebd95a375584: Pull complete
7214066b9e5c: Pull complete
49de0690a3af: Pull complete
Digest: sha256:fe65e29d780e61bc7a8529a527676628231c7713aacdfc4887e6b6e4910d5e79
Status: Downloaded newer image for quay.io/microcks/contract-testing-demo:01
Example app listening on port 3001The application is available on your host http://localhost:3001 or on http://host.docker.internal:3001 when reached from the Microcks container running in the docker-compose network. That is the endpoint I’m using to launch an OPEN API SCHEMA test as illustrated below:
As explained in the scenario introduction, this test fails. Let’s check the Full results in Microcks to get details on the received responses from the tested endpoint as shown below.
You’ll see red crosses everywhere and also get validation error messages. In our case here, the pastries included in responses miss the mandatory status property and have an invalid type for the price property (a string instead of a number). As the foundational syntactic conformance is not valid, we’ll stop here and won’t even assess behavioral conformance.
This closes the first part of our scenario: where we demonstrated the ability of Microcks to detect syntactic conformance issues automatically by exploiting an OpenAPI Schema. No need to write code for that! Suitable schema rules are applied for each tested request.
Testing Version 02
Let’s now run version 02 of the application. Just launch a new container from the image tagged 02 and map it to the 3002 local port:
$ docker run -it -p 3002:3002 quay.io/microcks/contract-testing-demo:02
=== OUTPUT ===
Unable to find image 'quay.io/microcks/contract-testing-demo:02' locally
02: Pulling from microcks/contract-testing-demo
b964235f9f30: Already exists
947a9cb9218d: Already exists
1128b5439c0c: Already exists
d0ac8fc3cde2: Already exists
ebd95a375584: Already exists
f8182c8ad91e: Pull complete
e5a09a9a5fe3: Pull complete
Digest: sha256:0a2fa124f72b9fee045d8da266e56cda0c35a80b52488a900b00379f4fe939a4
Status: Downloaded newer image for quay.io/microcks/contract-testing-demo:02
Example app listening on port 3002This new version is available on your host http://localhost:3002 or on http://host.docker.internal:3002 when reached from the Microcks container running in the docker-compose network. We can now re-launch an OPEN API SCHEMA test but now specifying this new test endpoint:
Hooray! 🎉 This time it works! We can see that the pastries in responses are conformant to the schema as the status property is present and price is correctly represented as a number.
But wait a minute… Have a look at the Full results and check again the content of the responses. We requested a Millefeuille but received an Eclair Chocolat! Even if pastries are syntactically correct, the returned values make no sense regarding the different request parameters:
- We did not get the correct pastry details,
- Pastries returned in the list filtered by size don’t all have the same size,
- And the pastry price is not updated as we may expect it to be!
This is because these expectations are not syntactic rules and are not expressed within the OpenAPI Schema! They are behavior expectations that cannot be translated into a schema, so Microcks cannot validate them — at least using the OPEN API SCHEMA strategy 😉
That’s where Postman Collection enters the game! Of course, we can use them instead of OpenAPI Schema to describe examples that will be transformed by Microcks into mocks but we can also use them to hold tests. Postman Tests are script assertions that provide enough flexibility for specifying the behavior or business-level expectations regarding your API responses.
You’ll see in the Postman ecosystem many references to contract testing actually related to syntactic conformance checking. There’s even a Contract Test Generator that’s able to translate an OpenAPI schema to a series of Postman tests. Microcks usage typically avoids this initial test generation step and — above all — avoids all the maintenance and de-synchronization issues when it comes to evolving the OpenAPI schema. (And this is true for all the approaches where syntactic tests are expressed as code like in Pact). You let Microcks handle syntactic tests and focus on writing scripts for checking behavioral/functional conformance. This also leads to a better separation of concerns IMHO.
Here’s below the Postman script we may write to ensure that getting the pastry details actually returns the pastry with the requested name. We collect the expected name from variables and then set the expectation that it is the same as the response name:
We can easily do so for the operation that allows listing the pastries having a specified size. We’ll put here the expectations that each pastry in the list will have the expected size:
Finally, here are the testing assertions for the operation updating the price of a pastry: we ensure here that the price provided in the request has been applied and is present in the response:
The resulting Postman Collection has been exported and stored into the demo GitHub repository. The next step is to import it into Microcks so that it will be able to use-it as a testing artifact. For that, you’ll have to create a scheduled importer using the https://raw.githubusercontent.com/microcks/api-lifecycle/master/contract-testing-demo/apipastries-postman-collection.json URL.
One important thing to notice here is that we’re going to use the Multi-artifacts support feature of Microcks and for that we must absolutely tick the Secondary Artifact checkbox illustrated below:
That way Microcks will try to merge Postman Collection information into the already existing API Pastries — 0.0.1. You just have to hit the Next button a couple of times and then head up to the API details page.
Some things changed from our latest visit:
- There’s now an additional Postman Collection contract attached (link added in
1.7.1release of Microcks), - The conformance score has bumped to
82.5%which is the maximum for this API (see Conformance Metrics for more details), - You may also notice that the number of samples attached to the “Get pastry details” operation has changed from 2 to 3. We’ve also illustrated here BTW that we may use a Collection to provide additional sample datasets.
From here, we can launch a new test on our http://host.docker.internal:3002 endpoint, choosing this time a POSTMAN runner:
After a few seconds, you’ll get access to the Full results and may check that our behavioral/functional expectations have not been met. The tests are clearly failing:
This test failure closes the second part of the scenario. We saw that OpenAPI Schema validation can be performed automatically by Microcks to ensure syntactic correctness. However, OpenAPI Schema alone is not good enough for detecting behavioral rule drift. This is where Postman Collection can be used in addition to OpenAPI for adding expectations on behavioral/functional rules. Microcks allow using both artifacts with a uniform approach to run the different levels of tests.
Testing Version 03
For our last round, we have already everything set up in Microcks and we just have to launch Version 03 of the application. Run a new container from the image tagged 03 and map it to the 3003 local port:
$ docker run -it -p 3003:3003 quay.io/microcks/contract-testing-demo:03
=== OUTPUT ===
Unable to find image 'quay.io/microcks/contract-testing-demo:03' locally
03: Pulling from microcks/contract-testing-demo
b964235f9f30: Already exists
947a9cb9218d: Already exists
1128b5439c0c: Already exists
d0ac8fc3cde2: Already exists
ebd95a375584: Already exists
a4c06ffbc0a3: Pull complete
5469e61c38e4: Pull complete
Digest: sha256:23ca84d0b657542e642fe65364e901312853b482799947cecfd68607b8b1b54c
Status: Downloaded newer image for quay.io/microcks/contract-testing-demo:03
Example app listening on port 3003To ensure we’re not facing regression, we may re-launch an OPEN API SCHEMA test on this new http://host.docker.internal:3003 endpoint shown below:
Test results are still green! ✅ Looking into the details in Full results also let us think that we’re on our way…
Let’s validate that with a new test on http://host.docker.internal:3003 endpoint, now using POSTMAN test runner this time:
And here we are! No Postman Collection testing assertion has failed and behavior is really as expected! 🍾
This closes the last part of our scenario where we demonstrated that full contract conformity is in fact achieved using both OpenAPI Schema and Postman Collection artifacts. Each artifact is focused on its comfort zone:
- Syntactic conformance checking for OpenAPI Schema,
- Behavioral conformance checking for Postman Collection tests.
Microcks actually provides a uniform approach to the test process whatever the kind of artifact.
Wrap-up
We demonstrated via this simple use-case and scenario that contract testing is actually made of two parts: syntactic correctness testing and behavioral testing. We saw that OpenAPI Schema alone is not sufficient for detecting behavioral rules drift. Postman Collection test scripts shine complementing a schema by adding expectations on behavioral/functional rules. I think it also brings a nice separation of concerns and allows efficient collaboration between actors with different backgrounds.
Whilst we’re using different specifications or tools, it’s important to have a consistent and uniform approach to ease the process of running the different levels of tests. This point is even more important in the multi-protocol / multi-API styles world that is in front of us. This is exactly what Microcks target: providing a universal approach to mocking and testing for all kinds of API (think AsyncAPI, GraphQL, gRPC, …) and different levels of testing (technical, syntactical, functional, …)
As a bonus, I just wanted to illustrate two additional features of Microcks:
- The test history contains details on exchanged data with the different endpoints,
- The CI/CD integration allows you to automate everything from your favorite pipeline technology!
Thanks for reading! Do not hesitate to react and share your thoughts! 💭
