Intro to PACT for .NET Core: API contract testing
Recently i came a across a situation where i had to explore contract testing using PACT framework.
For some background, the account that i was working for had already implemented contract testing in their micro-services. They were following schema based contract testing approach.
In short, schema based testing approach involves matching of the schema that consumers of an API are using against the said API provider. The example and images below will illustrate schema based testing using a small example scenario.
Imagine a microservices system with 3 consumers (A, B and C) for a Provider API. Now all three consumers are expecting a certain fields for their use case from the Provider API. As long as Provider is able to maintain the contract expectations of the consumers, things will be fine.
In schema based contract tests, the testing framework is responsible for getting the schemas of all the consumers and it also holds the current schema of the Provider service. The next step is the comparison which is show in images below.
In diagram above, a success scenario is shown in a schema based contract testing. All the tests are success as the expected fields by all the clients are available in provider’s schema.
The above diagram shows a failure case. The failure is because the Consumer C is expecting the role field which now has been removed from the Provider API schema.
Everything till here looks good, we are able to capture failures, then why we explored another options ?
Its because in the approach above, the contract test pipeline was running on development environment and the schema was taken from a currently deployed service in development environment. This means that the failure will only be caught after the breaking change has been introduced in the Provider API repository and deployed to certain environment. In short failure detection was late.
As you can see above, the failure detection is after the build and deployment step were done.
We wanted the build step itself to fail so that an early detection of the breaking change is caught before it can make make any mess. Something like below.
How we achieved the above using PACT framework ?
I will focus on how i implemented consumer driven contract testing using PACT in .NET Core API. I will not go into the very details of what PACT is, as that is explained much better at https://docs.pact.io/.
We will dive into the code with below services setup:
- Student service (PactNet.Provider): Provider API
- Report Card Service (PactNet.ConsumerOne): Consumer API
Student service exposes below API:
I had the solution structure as below.
Consumer side tests:
The very first step is to create a unit test file and initialize pact builder as shown below
Next we have to write test and setup scenario. Here below, we are instructing pact builder to return a specific response when it gets a specific request on the mock server (ctx.MockServerUri).
The success test run will look like below
Once above test run, 2 things will happen.
First, the logic inside the StudentClient class will be tested to make sure the request that is dispatched from the system to the provider service should contain all the required request attributes that are defined in the unit test.
Second, a Pact file will be generated with the name ConsumerOne — Student API.json. The naming of the file can be defined at the time of pact builder initialization.
This pact file needs to be passed to Provider API so that it can be accessed inside unit test, for simplicity here, we have included this file as part of Provider API’s unit test project itself. However, there is much better way which provides many other features, that is called pactflow(https://pactflow.io/ ). It offers free plan as well which can be used to small or test projects.
The pact file will look something like below. This file contains, among other details, the request structure that ConsumerOne(Report Card API) will be sending and response that it expects from the Provider API
Provider side tests and verification of pact file:
Once we have the pact file with us, we can use this pact file to validate it against the Provider API.
Provider side tests take a little more effort than the consumer side tests, as it requires more setup.
Essentially what happens is that PACT tries to fire up an actual instance of the provider service and test the same request as mentioned in the pact file against this fired up instance and then matches the response to validate.
One must be wondering then, that actual instance will require other stuff as well, like a database, any third party APIs involved and many more. So what will happen with all that ? This is true, however, in case of Pact provider tests, all those things can be mocked. For database, if we do not want to mock, we can use in-memory databases as well. At the end, our target is to test the logic that resides in our code base that can possibly affect the nature of contract.
Mocking is done in the same way we mock normal unit tests. Hence, the provider contract tests will also be running like unit tests only. Provider unit tests structure looks like below.
In the StudentApiFixture class we are trying to setup and start a host
In ProviderStateMiddleware class we are trying to setup the state of the system. It is to note that using Pact we can write complex test scenarios as well. For example lets say if student address anything other than Delhi then, consumer is expecting an additional field called City. These kind of scenarios can be handled by defining the state of the Provider API using state middlewares like below.
In the below middleware, we are adding a middleware that will add a student with id 1 in the system. If you correlate, then code (line #28) below is the same sentence that is mentioned in the pact file, provider state section.
Next step is to define the TestStartup(can be named anything) class. This class contains all the code that our Provider API needs to get started.
It is to note that an actual production application may contain various other services and middlewares like, logging, analytics, health checks etc, which does not affect the code logic. However, since our purpose is limited to testing the code that can potentially impact the contract side of things of the application, so we can ignore all other things, to make our tests lighter. This should be subset of the actual Startup.cs file in the Provider API. In short, just enough for our tests to run.
Also, since ours is a small test application we are not mocking the methods defined in IStudentRepo class, but we can mock them whenever required. The idea is that Pact test should be able to run in isolation, not depending on any actual service or any environment.
Finally the test case. In the test case, instruct Pact verifier about which fixture class to use for setting up and running the host (line 16) and along with the path of the Pact file that was generated by the consumer (line 39). Pact verifier take all the required information about host and pact file and finally runs Verify() method.
The test run will look something like below.
Failure Scenario
To test out failure scenario, we will try to change the pact file manually and mimic a scenario such that the consumer is expecting another field called “classId” (line 26) which, at the moment, our provider API does not supports. After manually changing the pact file the response expected by consumer service will look like below:
After this, if we try to get this pact file verified by our provider unit test, we will get a failure result indicating why the test failed. The failure message ($ -> Actual map is missing the following keys: classId) below indicates the field name that was not supported by the provider API but was expected by the consumer service.
Now, since the contract verification is happening at unit test level, we were able to handle the breaking changes detection at build level only. Even better, we can use pre-commit hook so that we get information on breaking changes at development level and faulty code is never committed to code repository.
For reference, the code repository being is discussed is available on github link: https://github.com/ajaysskumar/pact-net-example
Thanks for reading through. Please share feedback, if any, in comments or on my email ajay.a338@gmail.com.
