How We Do Contract Testing at Teachable

Published in

Teachable

8 min readSep 7, 2023

In my previous post, I walked through how I built my first microservice. As we build more microservices at Teachable, we encounter new challenges, especially as the service footprint grows. Specifically, running end-to-end integration tests becomes increasingly difficult.

Integration tests help us validate the successful interaction between two or more components given some set of expectations or “contract.”

Let’s walk through how integration tests are typically done. Given a Consumer A that is dependent on an API defined by Provider B, we would spin up both services locally (Dockerized of course) and then run the test. This gets complicated as the number of dependent services scale up.

Another way we could run an integration test is against staging environments. This can be slow to iterate on or unreliable due to specific dependencies being down or other flaky issues.. Multiple points of failure implies decreased developer velocity and slow iteration, not to mention how expensive it is to maintain large testing infrastructure. As we stated previously, an integration test helps validate a set of expectations, which is captured perfectly in the concept of a contract.

With consumer-driven contract testing, the consumer defines a contract that the provider should adhere to. Before I dive into the details, there are a few key concepts to take note of. I’ll use the Reddit app, Apollo, as an example:

Consumer -> The component that is dependent on functionality or data of another component via HTTP.

Apollo is a consumer of the Reddit API.

Provider -> The component that provides functionality or data for other components to use via its API.

Reddit is a provider to Apollo since it serves Reddit data through its API.

Interaction -> This is essentially a test created by the consumer that will run the expectation against a mock server, provided by the pact library, to generate the resulting pact contract.

When Apollo fetches Reddit posts and comments it expects a certain response schema.

Contract -> This is the “pact” contract JSON file that gets generated during an interaction. This is the file that will get published to the PactBroker.
Pact Broker -> Allows you to share your pact contracts / verification results between services. These verification's are pivotal to ensure the contracts are being adhered to.

At Teachable, we have ~12–15 microservices and that number is constantly growing. We might be a bit early for contract testing, but having the right processes and workflows in place now will prove to be invaluable later. The goal was to build the contract testing pipeline for two services and demo the benefits to the rest of the engineering team.

Generate the Contract

Let’s begin by defining our consumer and provider:

Provider -> School Plan Service is our Go application that handles data related to a school’s Chargebee subscription (i.e. plan information, permissions, etc.).
Consumer -> Fedora is our Ruby on Rails monolith that powers most of Teachable and serves as a consumer of many smaller services.

Given that we are following consumer-driven contract testing, Fedora will be responsible for creating our pact contracts via rspec tests. These interactions look like so:

# Creates a mock consumer service on localhost:9002 that will respond to 
# our queries over HTTP as if it were the real "Fedora" consumer.
Pact.service_consumer "Fedora" do
  has_pact_with "SchoolPlanService" do
    mock_service :school_plan_service do
      port 9002
      host "localhost"
    end
  end
end

# Define the rspec that will create the pact file.
RSpec.describe "School Plan Service", pact: true do
  describe "GET school plan subscription data" do
    let(:consumer_expectation_mock) {
      {
        plan: {
          status: "fun status",
          tier: {
            name: "fun plan",
          },
          plan_attributes: {
              fun_attribute_1: "fun attribute"
          }
        },
        school_id: 1,
      }
    }

    # This is the interaction between our provider and 
    # our consumer expectation (ie. consumer_expectation_mock).
    before do
      school_plan_service
        .given("A School with ID 1 and a fun plan exists")
        .upon_receiving("a request to fetch school plan subscription data")
        .with(method: :get, path: "/api/schools/1", query: "", body: nil)
        .will_respond_with(
          status: 200,
          body: consumer_expectation_mock
        )
    end

    it "returns the school plan subscription data" do
      response = RestClient.get("http://localhost:9002/api/schools/1")
      expect(response.code).to eq(200)
    end

Running this rspec test produces a JSON file that will serve as the pact contract:

{
  "consumer": {
    "name": "Fedora"
  },
  "provider": {
    "name": "SchoolPlanService"
  },
  "interactions": [
    {
      "description": "a request to fetch school plan subscription data",
      "providerState": "A School with ID 1 and a fun plan exists",
      "request": {
        "method": "get",
        "path": "/api/schools/1",
        "query": "",
        "body": null
      },
      "response": {
        "status": 200,
        "headers": {
        },
        "body": {
          "plan": {
            "status": "fun status",
            "tier": {
              "name": "fun plan"
            },
            "plan_attributes": {
              "fun_attribute_1": "fun_attribute_1"
            },
          },
          "school_id": 1
        }
      }
    },
  ],
  "metadata": {
    "pactSpecification": {
      "version": "2.0.0"
    }
  }
}

A few things occurred during the rspec run:

The interactions were run against a mock service provider that’s defined on localhost:9002. This is provided by the pact-ruby library.
The mock service returns the expected response and creates the pact file as JSON. If you have worked with VCR to record HTTP interactions to playback, it’s the same concept.

So where are we at this point in time?

We have a pact contract defined
The Provider does not know about the new expectation
Pact Broker does not know about the contract

Publishing the Contract

Now that we have a contract, we need to figure out how to get it to the provider, so it can run the contract expectations against itself. One way to do this is to just hardcode the path of the generated pact file in the provider code every time you need to generate a new pact. Fortunately, this manual process can be automated using the Pactflow Broker. The Pactflow Broker is a service that allows teams to interact with pacts and verification's in an automated way. The Docker Pact Broker is a nifty tool we leveraged within our PR pipeline to publish a new pact to the broker whenever expectations are created or updated. The published pact would also be tagged with the first 7 characters of the latest commit SHA to ensure uniqueness between published contracts.

Provider Verification

During local testing, we hardcoded the path of the generated pact as an argument to the provider verification test as follows:

_, err = pact.VerifyProvider(t, types.VerifyRequest{
         ProviderVersion: os.Getenv("GIT_COMMIT"),
         ProviderBranch: os.Getenv("GIT_BRANCH"),
         ProviderBaseURL: fmt.Sprintf("http://localhost:%d", port),
         PublishVerificationResults: publishResultsBool,
         ConsumerVersionSelectors: getSelectors(),
         PactURLs: []string{"/full/path/to/pact-file.json"},
         BrokerURL: os.Getenv("PACT_BROKER_BASE_URL"),
         BrokerToken: os.Getenv("PACTFLOW_TOKEN"),
         EnablePending: true,
         IncludeWIPPactsSince: "2023–01–01",
         StateHandlers: types.StateHandlers{
           "A School with ID 1 and a fun plan exists": func() error {
               return nil
         },
     },
 })

Before this verification is run, we start our provider server on localhost and the code block above will verify that the current state of the provider (on the provided GIT_BRANCH) adheres to the pact contract located in the Pact URL. If verification succeeds,, we can decide to publish the results to the broker via PublishVerificationResults. This unblocks the consumer so they can proceed to deploy their changes. If the verification fails, we communicate with our consumer team on next steps. We have successfully, albeit manually, completed an end to end pact contract test.

Pactflow Webhooks

A few key concepts to keep in mind for this section:

Webhooks -> An HTTP push API that allows you to trigger an HTTP request when a certain event occurs (ie. pact changes published to broker)
Pact Matrix -> A table created within the Pactflow Broker that contains all the consumer/provider pact versions that have been tested against each other. It includes metadata such as branch names, environments, published date and, most importantly, the verification status
can-i-deploy -> A command provided via the pact-broker CLI tool that allows you to query the pact matrix the compatibility of consumer/provider versions

In the previous section, we hardcoded the Pact URL within the provider to perform the test. This is not ideal since we would have to update this every time there’s a new pact. Moreover, it makes it difficult for the consumer to quickly determine the verification status of their latest pact with the provider.

That’s where webhooks come into the picture. When there’s a newly-published pact or changes to an existing pact, we can trigger a provider verification workflow that will update the verification status on the Pactflow Broker and enable the consumer to quickly see those results. There’s also a webhook-less workflow where the consumers CI/CD pipeline checks out the provider codebase and runs the verification as part of the consumers test.

In our case, we are listening to the Contract Requiring Verification Published event. The process is as follows:

Consumer team updates the rspec containing the pact interaction in Fedora
They publish the new contract to the pact broker via pact-broker CLI.
The Pactflow Broker triggers the Contract Requiring Verification Published event hitting the github /dispatch api for our provider repo.
A GitHub workflow is triggered in the School Plan Service (provider) that verifies the new contract against the provider and publishes the verification status.

You might be wondering, does the consumer need to manually check the verification status of the latest pact? And this is where the can-i-deploy command comes in. We can couple the can-i-deploy command with the publishing of the new pact. This will look like so:

Publish the new pact to the Pactflow Broker
Provider verification will run via GitHub workflow and publish the status asynchronously.
The consumer pipeline calls can-i-deploy to get the verification status of the published pact. This will wait for the provider workflow above to finish and query the pact matrix for the resulting status. Note that you can configure the can-i-deploy step to poll for the verification status for a certain period of time (in case the webhook triggered provider workflow takes a while).
If can-i-deploy succeeds, you can deploy the changes.
If can-i-deploy fails, wait for the provider to implement the new expectations. Take a coffee break while the provider team gets busy!

Conclusion

This covers the major pieces of contract testing at Teachable. I didn’t describe certain features in detail because they deserve a separate post. Things like the pact matrix, pact versioning, and work-in-progress pacts are key to ensuring a robust pipeline.

The Pactflow documentation is fantastic and I was able to successfully create a working pipeline based on their resources. Contract testing might not work for all workflows and it’s important to understand when it does or doesn’t make sense.

Finally, this may not be the most efficient or perfect pipeline, but here’s what our consumer side pipeline looks like.

P.S. Special thanks to V who initially championed contract testing and gave a great presentation to the engineering team.

References: