Introducing OpenAPI Comparator: an open-source tool to detect breaking changes in your API
Protecting your API against changes that could impact your clients is one of the biggest challenges when developing an HTTP API. If you introduce breaking changes in your API, then you expose your clients to major issues. A stable API is a key factor in client satisfaction, and, ultimately, business success.
But there’s a catch, clients' needs can change over time, which often requires evolving the API itself. For instance, adding more options to act on a given resource, or removing some other options to keep the overall API simple.
Preventing breaking changes becomes harder as your API grows. With more than 250 endpoints, Criteo’s API has reached a point where manually detecting breaking changes is no longer sustainable. That’s why we looked for an automated solution. Let’s have a look at what we’ve built.
What is a breaking change?
In a nutshell, a breaking change is a change in your API that could cause failures in the applications that consume your API.
Some breaking changes are easier to identify than others. For example, the removal of an endpoint is considered a breaking change in most cases, as it will break the clients’ integration. But some other changes may be harder to catch, such as changing an optional value in an enum or query parameter to required for example.
But not all changes are breaking changes. Adding a property in a schema or adding an optional parameter should have no impact, as the applications consuming the API should not require any update to run.
Some other changes may not be reflected in your API contract, such as behavior changes. These breaking changes are harder to detect and will not be covered in this article.
Defining which change should be a breaking change or not mostly depends on your API. An early-stage API, whose stability isn’t the priority at the moment, may have a different definition of breaking changes from a longstanding API.
To give you an example, you can find a non-exhaustive list of changes that we support at Criteo here.
We use the OpenAPI Specification standard (OAS) to write our API contracts. It is by far the most advanced and supported standard for describing HTTP APIs. You may have heard of it under the name Swagger Specification, which is its former name before Smartbear® donated the standard to the OpenAPI Foundation.
Examples of breaking changes
Let’s go through the update of an enum value and see how this looks in an OpenAPI Specification:
Our starting point would be the following animal-type query parameter definition, whose value can be either cat, dog, or panda:
parameters:
- in: query
name: animal-type
schema:
type: string
enum: [
cat,
dog,
panda
]Let’s say that our API is updated, and the query parameter animal-type no longer accepts the value panda.
An application consuming the API may expect the value panda to still be valid. But as the API has changed, such value isn’t accepted anymore, and an error will be returned:
{
"traceId": "56ed4096-f96a-4944-8881-05468efe0ec9",
"type": "validation",
"code": "validation-error",
"instance": "/clinics/1234/pets",
"title": "Unknown enum value",
"detail": "Unexpected enum value: panda"
}Detecting changes, but where?
Criteo’s API is following an industry-standard design. It is broken down into several sub-services, each responsible for a sub-part of the API. An API Gateway is placed in front of those services as a single entry point for the API. Each call to the API first reaches the API Gateway which forwards the call to the right underlying service.
The Gateway knows which service to redirect the call to thanks to a routing configuration, coming from our service registry.
If you want to know more about how our API’s design has evolved over time, you only need to read this article: Building Criteo API, One Step at a Time.
The service registry is responsible for building the routing configuration. Each internal service has its own routing configuration and OpenAPI Specification contract, which are aggregated and exposed by the service registry. It seems like a good place to detect breaking changes.
Detecting changes, how?
The routing configuration and the OpenAPI Specification of each endpoint are stored in a git repository. This allows us to audit the contract change proposed by our internal teams and easily see the history.
In addition, a pull request for a configuration change triggers automated checks to ensure the validity of the routes and OpenAPI Specifications.
With such a process, every update of the external API is versioned, automatically validated, and manually reviewed.
One of the automated checks ensures that no breaking changes have been introduced. The check compares the aggregated OpenAPI Specification in the repository to the one in production. Each detected difference is categorized. Some of those types are considered breaking, while others are not. If differences are detected, and we consider at least one as a breaking change, then the pull request is rejected.
Being able to efficiently compare OpenAPI Specifications is a key component of such a system, this is where OpenAPI Comparator comes in.
OpenAPI Comparator to the rescue
As most of our stack is written in .NET, we were looking for a C# library able to compare OpenAPI Specifications. At that time, one project caught our eye: openapi-diff, an open-source project developed by the Microsoft Azure team. Unfortunately, it was not available as a NuGet package (the .NET package manager). And, more importantly, openapi-diff only works on OpenAPI 2.0, which was superseded by OpenAPI 3.0 in 2017. We decided to fork the project internally and migrate it so that it could compare OAS in version 3.0.
Now that the migration is over, it was only fair to share the tool with the community: this is how OpenAPI-Comparator was born, a C# library able to compare OpenAPI Specifications in version 3.0, available as a NuGet package. The library can detect many kinds of differences, which is useful when comparing two APIs or two versions of the same API. And for each detected difference, additional metadata is returned:
- An explicit description of the change
- Its location in the old and new specification
- A documentation link of the violated comparison rule
- A type, describing whether the change is a removal, an update, or an addition
- The severity of the detected change could be either Info, Warning, or Error
The comparator has a pretty simple API. It corresponds to a single method taking two OpenAPI specifications to compare as arguments:
IEnumerable<ComparisonMessage> OpenApiComparator.Compare(
string oldOpenApiSpec,
string newOpenApiSpec);The tool also has a command line interface, as well available as a NuGet package:
openapi-compare --old path/to/old-spec.json --new path/to/new-spec.jsonBased on the list of detected changes returned by the comparator, we can easily identify whether breaking changes have been introduced in the API.
Conclusion
When a client starts using your API, an implicit agreement is settled between you and your client: the same version of an API should not be updated with breaking modifications. Protecting your API from breaking changes is a strategic investment that should not be underestimated.
This task can be tedious and error-prone, and this will become more true as your API grows. This is why at Criteo we have designed and implemented an automated verification process, that compares a potential update of our API with the one currently released. If any breaking change is detected, then the update is rejected.
Automating the breaking change detection has resulted in a major productivity and efficiency gain. And we’re excited to get a chance to help the larger community of API builders by open-sourcing the OpenAPI Comparator (listed on openapi.tools — thanks, Darrel Miller 🙏).
Did you enjoy the article? Would you like to work on creating tools like this and helping communities? We love open-source initiatives! Check our positions.
