Microcks and Pact for API contract testing
This is a topic I had in mind for a long time… But sometimes life is hard with procrastinators: it keeps sending you signs that you should tackle and tick this item from your ToDo.
As Microcks now occupies most of my working days, I keep having this question from users I meet during events or on community calls: “How Microcks compares to Pact regarding contract testing?”. Even though I had some opinions and feelings on it for quite a while, I know I had to study more to be more accurate and impactful when answering this…
The trigger happened a few days ago as I was watching John Wick — Chapter 2 movie. To be clear: it was not the impressive number of Keanu Reeves victims that inspired me 🫣 It was rather the main plot of the movie — about the honoring of contract, the reimbursement of debt — and this artifact below the protagonists use to seal with their blood such business.
In short: signing a contract with John Wick is like doing a pact with the devil. I absolutely had to write this post. Make my ideas clearer about Pact and write this down: I owe this to the Microcks community! (otherwise, I have to fear Baba Yaga, for those who’ve seen the movie 😉)
So this is my attempt to clarify the similarities and differences of using Microcks and Pact for API contract-testing. As a starting point, I decided to use Pact’s own definition of contract testing I like pretty much:
Contract testing is a technique for testing an integration point by checking each application in isolation to ensure the messages it sends or receives conform to a shared understanding that is documented in a « contract ».
I have tried to be very fair and follow the same approach for describing the features and principles behind both solutions. However, I cannot resist sharing my point of view and opinions at the conclusion of this post 😉
Disclaimer: I’m not pretending to be a Pact expert at all. I spent a decent amount of time going through the documentation and experimenting with it but I’m still learning. May you find any inaccurate statement or think some nuance and complement should be brought here, please let me know and I’ll be happy to revise the post.
Microcks way of contract-testing
What? When?
Contract in Microcks is a specification artifact: an OpenAPI or AsyncAPI file (referencing JSON or Avro schemas), a GraphQL schema, a gRPC/Protobuf schema, a SoapUI file (referencing WSDL + XSD) or a Postman Collection (referencing examples). You can also use multiple types of specification artifacts to complement contracts with example datasets or additional testing rules (eg. if you want to test the behavior of your API through scripting).
Microcks militates for those contracts to be the source of truth that is actually used when documenting and communicating the expected intent of an API or Service. The fewer artifacts you have the lower the risk of synchronization issues and drift! So Microcks promotes reusing well-known and established standards.
In the API Full Lifecycle according to Microcks vision, contracts (and attached examples) are written first. These are the only elements required to produce full mocks and test suites for later developments.
Who?
Microcks fosters those contracts to be written and owned by the provider of the API, the Event or the Service (depending on your API-ish, EDA-ish or SOA-ish background 😉 ). We tend to think that this is crucial to avoid misunderstanding and false interpretations of exchanged messages and intended behavior. If you let consumers do this work, they’ll probably have their own biases and interpret things the way that suits them best.
Furthermore, we also think that it’s crucial to reduce the translation mismatch that can happen when business requirements are transformed into software assets by developers. Hence, keeping things simple, and sticking to descriptive standards that don’t require introducing code or Domain Specific Language is a way to allow Business Line experts and Developers to collaborate; eliminating the drift risk.
How?
Contract testing in Microcks is global schema conformance testing. It is about replaying the sample requests against a real-life test endpoint and ensuring that received responses are conformant to the schemas we have found in specification artifacts. That means that you don’t need to write specific assertions (using code) to check that your endpoint actually produces a valid payload, including the correct response code or headers.
Contract testing in Microcks is also interaction testing in the sense that Microcks is testing against real endpoints. That means that when you’re testing your application with Microcks, you’re evaluating it as a real-life consumer, going through the different layers of protocol serialization, network connectivity & serialization, etc.
In 10 words and a drawing
Microcks is provider-driven contract testing at the protocol level.
One important thing to notice here is that API provider and API consumer timelines are loosely coupled as consumer(s) will be able to start working in parallel as soon as your OpenAPI specification (ie. the contract — it could be some other things in Microcks) is available. Also, the provider does not depend on consumers to fully validate its application endpoints.
Pact way of contract-testing
What? When?
Contract in Pact — or simply a pact — is a JSON file describing the interactions between one provider and one consumer. Interactions are a set of request/response pairs, describing each message in terms of expected response code, headers, and body payload.
Pact JSON files are not typically handwritten, they’re generated by tooling from some Unit Test code you have written to describe your expectation. This means that you write the request you send and the response you expect using specific Pact libraries corresponding to your programming language of choice. When running the test, the Pact libs generate the Pact file for you.
In the API lifecycle according to Pact, contracts are then generated and come after you have previously written Unit Tests in a specific language, based on the assumptions you have about what the API provider should be.
Who?
As a consequence of the previous point, Pact encourages those contracts to be written and owned by the consumer of the API, the Event, or the Service. Pact promotes this approach as it allows the one writing pact to focus only on the parts of the communication that are actually used. This lowers the “burden” of writing tests and protects the consumer from breaking changes on parts it doesn’t use.
Because pacts are actually generated from code, Developers (or Testers that write code) are actually the ones you are writing and maintaining Pact files. The Unit Tests that allow the generation of such contracts are typically unreadable from Business Line or Functional experts.
How?
Contract testing in Pact is scoped conformance testing with assertions. Like in Microcks, it is about replaying the interaction messages against a real-life test endpoint and verifying that received responses satisfy the expectations set in the Unit Test and translated in the pact. That means that all the syntactic elements of communication should be explicitly verified: the payload, the headers & the response code in a consistent way throughout your test cases. Semantic (functional) aspects can also be verified and it may lead to a slippery slope.
Contract testing in Pact is realized using two different approaches. For HTTP based APIs, the Pact verifier issues real request against real endpoints; however for Event driven APIs, it takes the place of the intermediary (eg. the broker) and rather test the handling of the message at the code level. In that case, it bypasses the protocol serializations, network connectivity and serialization stages that make a comprehensive interaction.
In 10 words and a drawing
Pact is consumers-driven contract testing at the code level.
Things to notice here is that the API provider timeline is coupled with the one from API consumers. The API provider cannot fully verify its application before collecting Pact files from its different consumers. In fact, the more pacts the provider collects the more chance of having a comprehensive test suite it has — without any real certainty as each consumer is focused on the part it used.
One thing that intrigues me here — and so I put some question marks on the drawing — is how the consumer gets the initial raw idea of a consumer’s capabilities for delivering the expected API. People usually see specification artifacts as a way to convey API documentation but this look,s like a chicken-and-egg situation here… Also when standard specification like OpenAPI is a requirement — because of public API for example — what’s the real source of truth for the API contract? The Unit Tests you write your expectations to? The Pact files that are executed by the Pact verifier? Or is it the OpenAPI spec available to everybody? I’d really like to know how people are coping with synchronization.
Similarities and differences
As there’s already a lot of text in this post, I decided to sum-up this part in a simple table. This is much more detailed material you can decide to dig into later on and jump to the conclusion right now if you want 😜
Opinions
Now that we’ve gone through the main principles behind each solution, it’s time for some conclusions regarding the drawbacks and benefits of each approach. Of course, it will be opinionated but please remember my disclaimer in the introduction and please help me if you think it needs some nuances.
Let’s reverse the previous order and start with Pact. The main benefit I see in Pact is in its scoped conformance testing approach: each consumer focuses only on the parts it uses and sharing pacts with the provider allows this one to clearly spot affected consumers when introducing a breaking change. However, this came with a major drawback to me: a provider cannot rely on consumer-provided test suites to be certain to have a comprehensive way to validate its API implementation. Moreover, in our modern world where proposing and delivering new features as fast as possible is crucial, an API provider cannot wait for its consumers to collect the contracts they expect. And when evolving into a global scale economy, it can certainly not be influenced by the expectations of all types of consumers.
Microcks approach is the opposite: it defends the responsibility of producing contracts (and the burden of producing a test suite) on the API provider’s shoulders only. When doing this the synchronization between a provider and a consumer comes very early in the development lifecycle and can lead to true parallelization of work. As we embrace distributed systems and micro-services more and more, each software component is potentially an API consumer and provider. We must absolutely decouple their lifecycle the most to keep frequent and frictionless releases.
A major difference between the two solutions also lies in the people who are supposed to write and own the contract. Microcks favors the use of specification standards that are accessible to non-coding people in order to foster the collaboration between functional experts and developers when designing API contracts — even if there’s still a lot to do on collaborative visual API designers but that’s another story 😉. On the other hand, Pact makes use of programming languages and thus targets developers. I see it as a high risk of drifting between business requirements and reality. As we’re now in the era of “API Products” that should hold real business value, I think it’s super important to reduce this risk the more we can.
However, having this provider-driven approach in Microcks can also lead to a certain risk of total unawareness regarding the way the consumers actually use an API and what could be the impact of a potential change.
This, in fact, reminds me of some debates we add at the turn of the previous SOA era to the current micro-services era about Services Design Principles. At that time emerged the Promise theory, a model first proposed by Mark Burgess in 2004 and covered in his book In Search of Certainty (O’Reilly, 2015), is a study of autonomous systems including people, computers, and organizations providing service to each other.
As the will for autonomous teams emerged, we tend to think that you — read “the API consumer” — cannot influence and place obligations on other teams and services because you don’t own them; they’re autonomous after all. All you can do is choose whether or not you want to trust the promises of capabilities they offer. In software, this theory is also relative to the way we — read “the API provider” — should build our system to be super resilient, super performant, and so on. Because we made a promise, we should absolutely deliver and commit to it!
Ok but… when to use what? And can they be used together?
Ok, ok. Don’t get me wrong: I’m not saying here that “obligations / consumer-driven” is the old way and “promise / provider-driven” is the new one. Nor consumer-driven contracts cannot help a provider to be sure it’s not upholding its promises. We’re reasonable people and know that IT trends need to be balanced and carefully assessed before being adopted. There are both many illustrations of majestic monoliths and use cases where micro-services architecture makes sense — just to cover the latest trend. As always, it depends and the situation matters.
That said, I tend to think that Pact is more suited to the “Obligation Service Design style” when used inside the same organization. This one may usually use a known and limited set of technologies — those where Pact language and HTTP support shines, consumers may have a real influence and proximity with API providers, and the business taint of API may be limited. In this scenario, Pact is helpful across team boundaries to find out what your users want and even adapt your view to meet their needs.
On the other hand, I tend to think that Microcks is more adapted to the “Promise Service Design style” faced in medium to large cross-departments, or cross-organizations API or service-based interactions. This situation particularly brings a whole set of API styles (REST, gRPC, GraphQL, Events, SOAP, …), technologies, and protocols (Kafka, RabbitMQ, WebSockets, NATS, Google PubSub) where the uniform and protocol-level testing approach of Microcks may drastically simplify things. In those situations, we tend to see a more service-first (or API-first) orientation that also favors more proximity with business and functional experts when defining contracts. Even if there’s still room for improvement in visual API designers, the use of standard specifications like OpenAPI is also a no-brainer to ease communication across larger boundaries.
Does that mean that the “provider-driven” approach fatally takes us away from the consumer’s experience and expectations collection? I don’t think so. Collaboration models have drastically changed since the old SOA days… Standardization communities, open product designs, community contributions, Pull-Requests, GitOps, and some other practices open access to API providers to a much broader audience (think about open banking or all the coming regulations). Contract adjustment suggestions are at the fingertips of consumers — and may be specified using Pact.
The beauty of things is that both solutions integrate easily within your CI/CD pipeline. So you can imagine using a global contract-driven approach encompassing both solutions on where you see interests. I’d be happy to get your feedback on such an experience if you set it up.
Let me know your thoughts! 🤔
