Detect Breaking Changes in Your GraphQL Schema Automatically for Each Pull Request

Api versioning is a lot of work and hard to get right.
One of the big benefits of GraphQL is that you can avoid Api versioning and instead grow your Api in a backwards compatible way. This implies that each change must be checked for backwards compatibility. In this blog post we automate this check for each pull request and integrate it into Azure DevOps using the GraphQL Breaking Schema Change Detector project.

What Is a Breaking Change and Why Should You Avoid Them?

GraphQL was designed with the idea that each client should only need to download the data it needs and so to avoid the over-fetching problem. The scenario that your Api is consumed by many different clients is baked into the founding idea. When you grow your graph, the old queries of your clients should still work. Otherwise, not only will all your clients have to be updated, they will also have to do it at the exact time you deploy the newest version of your Api. Determining what change is allowed and what will break your schema is often trickier than expected. For example, if you delete a value of an enum it depends on the usage of the enum to determine if this breaks backwards compatibility or not.

What Change Could Break Your GraphQL Schema?

Most of us generate our GraphQL schema code first.
This means that the GraphQL schema is generated based on our domain model. A change in the domain model will then lead to a corresponding change in the schema. The danger of this method is that an innocent looking change like making a mandatory field optional will result in a non-backwards compatible change to your GraphQL schema. A good idea to combat this is to have a unit test which generates a snapshot of your schema. The snapshot, containing the schema definition in text form, will be checked into source control and reviewed in the pull request stage. It is then the pull request reviewer’s job to make sure that the proposed change is indeed backwards compatible. An example of how to generate such a schema snapshot can be found in my latest blog post about getting started with HotChocolate 12.

Automatically Detecting Breaking Changes

Manual checks are not only error prone but also time consuming and thus expensive. So, let’s see how we can automate this using my open source GraphQL Breaking Schema Change Detector. There are similar projects out there like GraphQL Inspector or Apollo Studio, but so far none of them offer integration into Azure DevOps.

The breaking change detector will create a review comment inside your schema diff at the exact location where a non-backwards compatible change was made. On each update to the pull request the tool detects if the issues are already resolved and will update the comments accordingly.

Azure DevOps has the concept of pipelines. A pipeline defines a build job including its environment and can be triggered for each change on a pull request. Pipelines are configured using yml files. Below you can see a pipeline definition which will first install the .NET 6 sdk and then the “graphql-breaking-schema-change-detector” .NET tool. The tool is then executed with the path to your GraphQL schema snapshot as a parameter.

On line 16 we see that the access token of the build agent is injected as an environment variable. The access token is needed to report breaking changes as comments inside your pull request. For this to work the build agent needs the permission “Contribute to pull requests”. The corresponding security configuration can be found in the Project Settings.

The setup is also described in the readme of the project. A complete example including generating the schema snaphost can be found here.

Final Thoughts

Evolving an Api over time is hard. The graph nature of GraphQL gives us a lot of flexibility how to evolve the schema but with this flexibility comes the cost of ensuring that all changes are backwards compatible. This is where the GraphQL Breaking Schema Change Detector can help you, especially if you are using Azure DevOps. Support for Github Actions is in the making.