Consumer-driven Contract Testing using Postman

Implementing consumer-driven contract testing is a great way to maintain growing microservices stacks. It frees teams from getting blocked on API discrepancies when done regularly. Here is how Postman can help you do it better.

Kaustav Das Modak
Oct 24, 2018 · 10 min read
Contract tests ensure sanity and uniformity of interactions between services.

API behavior is typically described in documentation pages which list available endpoints, request data structures and expected response data structures, along with sample query and responses. These documentation are then used by people building systems that consume those APIs.

But, documentation written separately do not shield the consumers of the APIs from changes in the API. API producers may need to change the response data structure or rename an endpoint altogether to keep up with business requirements.

The onus of incorporating these changes then falls on the consumers of those APIs who have to keep checking the documentation for any changes. This model does not scale well. Consumers often end up with unexpected bugs because the response they were expecting had changed.

This is where having consumers assert on API contracts become useful. Instead of having API producers build a specification on their own, consumers of those APIs can set expectations by letting the producers know what data they want from the API.

An example scenario documenting steps that are usually taken in building consumer-driven contracts for microservices

API design then turns into a negotiation between what the consumers need and what the providers can give.

Make API contracts explicit and executable

Instead of documenting API behavior in a static page, producers can start by creating a specification for their API with any of the popular tools on the market, like RAML, API Blueprint, OpenAPI, Pact or Postman Collections. The last two, Pact and Postman let you implement consumer-driven contracts as a first-class concept. Pact further focuses exclusively on contract testing, while the Postman ecosystem packs a lot more power beyond that.

All of these formats allow you to specify details about your API’s behavior. They let you convey the intent of the API through endpoint names, descriptions, data types and data structure for requests and responses. They also support adding examples to each endpoint.

Once you have your specification in these formats, you can run tools that generate test code or directly send requests to a given service using the structure of endpoints described in the spec. The degree of flexibility you get varies with the toolset you use.

These specifications form the source of contracts for your service — the agreement between what the producer provides and what the consumers can expect. The underlying value of each of these toolsets is that they make your API structure explicit to those collaborating to build the service as well as to the service’s consumers.

Test contracts against multiple consumers

  1. You can run your test suite whenever there is a change in the specification and respond to changes in the API specification proactively, and,
  2. You can participate in the design process of the API as a consumer and let your needs be known before a contract is formalized.

In consumer-driven contract testing, the contracts are written and managed explicitly by the consumers. If the producer needs to make some changes to their service, like implementing a new feature, producers will get to know which consumers they broke just by looking at which consumer’s contract tests have failed. This frees API providers from worrying about accidentally breaking consumer apps or services every time they make some change to their service.

Independent service testing

End-to-end tests can be expensive and cumbersome to do for every change to the API. Producers and consumers can move at their own pace, have their own roadmaps and continuously deploy their changes. Consumer teams need to worry only when there is a failure in the contract tests. Coupling this with an updated documentation, generated automatically from the specification, make failures quick to detect without building up complex setups required to run end-to-end tests.

Using Postman for consumer-driven contracts

Further, you can add dynamic behavior to your requests with pre-request scripts and assert on responses with tests. The article on transitioning from manual to automated API testing in Postman runs you through details of these steps.

Couple all of these with Workspaces in Postman and you have executable, shared and collaborative contract collections ready for your entire team. You don’t need to manually share these details with your team members.

Let me run you through an example use case on how these come together to implement consumer-driven contracts.

A sample use case: Simple log retrieval service

The endpoint resides at /api/v1/logs. Sending a GET request to this endpoint is supposed to return JSON data in this structure:

{
"count": Number,
"entries": Array[Object]
}

The entries array will contain an object for each log entry. Their data structure will look like this:

{
"serviceName": String,
"timestamp": Number,
"description": String
}

Blueprint

You can access the blueprint collection for this example here.

Sample blueprint collection

Then, we add examples for the request. I had already added an example. Examples allow such blueprint collections to describe response data. They show up in documentation generated by Postman. Here is an example of response data for the default output of this service:

{
"count": 5,
"entries": [
{
"serviceName": "foo",
"timestamp": 1540206226229,
"description": "Received foo request from user 100"
},
{
"serviceName": "bar",
"timestamp": 1540206226121,
"description": "Sent email to user 99"
},
{
"serviceName": "foo",
"timestamp": 154020622502,
"description": "Received foo request from user 10"
},
{
"serviceName": "baz",
"timestamp": 1540206223230,
"description": "Activated user 101"
},
{
"serviceName": "bar",
"timestamp": 1540206222126,
"description": "Error sending email to user 10"
}
]
}

Each example has a name and specific request path. This is how it looks in the Postman app:

Adding example to sample blueprint collection’s request

With these in place, Postman generates web-based documentation for the blueprint automatically. Below is a screenshot of how the published documentation looks like.

Published documentation generated by Postman from the sample blueprint collection

All members who are part of the workspace in which this collection was created can view the documentation and access the collection, making it an excellent way of collaborating with other members. The service owners can then create a mock server in Postman based on this collection. The examples added in the request are sent as part of the mock server response.

Once a mock server is created in Postman, you can make requests to the mock service by using the same endpoint after the mock server URL that Postman generates for you. So, making a request to https://<mock-server-id>.pstmn.io/api/v1/logs will return the following response:

Response returned from mock server created from sample collection

Writing contract collections

For this example, let us assume there is only one consumer of this service. To keep things simple, our contract collection example will also have one request and it will assert only on the response data structure. A real-world contract would assert on the data structure as well as the data received in the response.

Consumer contract collection using tests to assert on response data

Here is the test script that the contract collection can use to test the data structure mentioned above. It uses the tv4 library, which is shipped as part of the Postman Sandbox:

// Define the schema expected in response
var responseSchema = {
"type": "object",
"properties": {
"count": {
"type": "number"
},
"entries": {
"type": "array",
"items": {
"type": "object",
"properties": {
"serverName": {
"type": "string"
},
"timestamp": {
"type": "number"
},
"description": {
"type": "string"
}
}
}
}
}
}
// Get response data as JSON
var jsonData = pm.response.json();
// Test for response data structure
pm.test('Ensure expected response structure', function () {
var validation = tv4.validate(jsonData, responseSchema);
pm.expect(validation).to.be.true;
});

The contract collection is published here. You can use the “Run in Postman” button on that page to load the collection in your Postman app and explore the test associated with the request.

Note the use of the {{url}} variable placeholder in the contract collection. When the service is in its early phase, consumers can use the mock server URL to make the requests. When the service has been built, the environment variable can be switched to point to a hosted instance of the service. This way, development of the consumer apps or services can happen in parallel without remaining blocked for the upstream service to be built.

Continuous testing

If you have an existing continuous integration system: You can export collection files and environments from Postman and run them from the command-line using newman. Refer to newman’s documentation for steps to set up continuous builds for Jenkins and Travis CI. Your build pipelines can be triggered every time there is a change in the specification or a new version of the upstream service.

Along with that, you can also run contract collections using Postman Monitors. Monitors can run periodically, making it an excellent tool to get a long-term overview of contract breakages.

Read more about Continuous Testing of APIs with Postman.

Organizing contract tests

A neat pattern of emulating this in Postman is to have a Setup folder as the first folder in the collection and a Teardown folder as the last folder of your collection. All contract tests then go as folders in between the Setup and Teardown folders. This will ensure that Postman will always run the requests in the Setup folder in the beginning of a collection run, then run the requests performing actual testing, and end with running all the requests in the Teardown folder.

We make use of this pattern heavily when writing the internal contracts.

Setup and Teardown folders in an actual contract collection used by the Postman engineering team.

This helps in grouping and organizing your tests neatly by abstracting out repetitive tasks in the first and last folders. Ideally, the API producer should provide a collection with the Setup and Teardown requests. Consumers can create a duplicate of that collection and add their contract tests.

The complexity of your contract tests will depend on your business case. One of the extra benefits of writing consumer-driven contract tests is that you can easily spot when a service gets too big or has too many dependencies by looking at the number of consumer contract collections that service has.

Overall, consumer-driven contracts help keep the surface area for testing microservices and negotiating changes to a controllable size.

Happy testing!

Better Practices

For individual engineers to the largest teams, Better…

Better Practices

For individual engineers to the largest teams, Better Practices is intended to distill knowledge from the Postman community. This is a place to learn about modern software practices together! Read more: https://medium.com/better-practices/introducing-better-practices-e9cf14cf0c88

Kaustav Das Modak

Written by

Looking for patterns in chaos.

Better Practices

For individual engineers to the largest teams, Better Practices is intended to distill knowledge from the Postman community. This is a place to learn about modern software practices together! Read more: https://medium.com/better-practices/introducing-better-practices-e9cf14cf0c88

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store