Easy Contract Tests with eslint-plugin-graphql
Note: this was originally published in 2017. While much of the tooling has changed, the high-level process described in this article is still valuable for many codebases.
Background
Here at PolicyGenius, we started breaking away from our Rails monolith in the middle of 2015, writing RESTful services to help us ship faster and support our growing line of product offerings. Shortly after this, we started moving our front-end off the monolith, adopting React and modern JavaScript tooling. At the end of 2016 we adopted GraphQL for one of our internal services, and are now using it for all of our products moving forward.
Among the primary benefits of adopting a GraphQL schema is the simplification of client-driven development. You can write one schema to meet the data requirements of all your client applications, whether for browser, mobile or salt shaker. Similarly, the same schema definition used for a single enterprise’s internal services can be leveraged for external client services, as companies like GitHub have done. A less-commonly known virtue of GraphQL is the ease with which one can write contract tests between these services.
The Problem
A long-understood best practice of a well architected microservice ecosystem is consumer-driven contract testing. In the June 2016 article “Consumer-Driven Contracts: A Service Evolution Pattern”, Ian Robinson describes common problems associated with Service-oriented architecture and prescribes a Consumer-Driven Contract pattern for building service communities. In short, it can be burdensome to change your API when you have multiple consumers relying on it. We can alleviate this pain point with an automated testing suite that ensures updates to the API do not introduce breaking changes to any of the operations performed by the client.
Robinson describes the three characteristics of a consumer-driven contract:
Closed and complete A consumer-driven contract is closed and complete with respect to the entire set of functionality demanded of it by its existing consumers. The contract represents the mandatory set of exportable elements required to support consumer expectations during the period in which those expectations remain valid for their parent applications.
Singular and non-authoritative Provider contracts are singular in their expression of the business functionality available to the system, but non-authoritative because derived from the union of existing consumer expectations.
Bounded stability and immutability A consumer-driven contract is stable and immutable in respect of a particular set of consumer contracts. That is to say, we can determine the validity of a consumer-driven contract according to a specified set of consumer contracts, effectively bounding the forwards- and backwards-compatible nature of the contract in time and space. The compatibility of a contract remains stable and immutable for a particular set of consumer contracts and expectations, but is subject to change as expectations come and go.
Here at PolicyGenius, we’d been talking about setting up contract tests long before we ever adopted GraphQL, but previously never had the time and resources to make it a priority. Fortunately this is an easy strategy to adopt in GraphQL land. Thanks to the strongly-typed GraphQL spec, we can statically analyze and check our queries against our schema as a part of our deployment pipeline for both our client and server applications. Apollo Data maintains eslint-plugin-graphql, a nifty little tool that handles the brunt of that work for us.
How to set up GraphQL contract testing as part of your CI build
Our strategy
We will create steps in our Continuous Integration (CI) builds for our client applications and our GraphQL server. At a high level, it looks like this:
Provider build:
- Before deployment to staging or production, pull down .graphql queries being used by our clients in the respective environment and validate that our changes will not break the client application.
- After deployment, write an introspection of our schema to cloud storage for the respective environment.
Client build:
- Before deployment to production, pull down the latest schema representation and validate our queries against that.
- After deployment, write a copy of all our .graphql files to the cloud. When there are multiple clients, it is helpful to put them in the same bucket under a directory specific to that project.
Requirements
To adopt our strategy, your client application needs to meet two reasonable requirements: First, write all of your queries in .graphql or .gql literal files. The benefits of doing this are well-documented and as far as I can reckon, there are no glaring drawbacks. Secondly, you must use eslint, at least for checking your GraphQL queries.
Publish your schema for client consumption
eslint-plugin-graphql checks your queries against a schema expressed either as JSON from an introspection query or (as of version 1.1.0) a schema definition document written in the GraphQL IDL. We recommend the former, since this can be leveraged for a lot of other awesome community-supported tooling as well.
On each successful deployment of your provider, generate a schema.graphql file. The way you do this may vary depending on your language of choice/library used on the backend. In our Rails project, we have a rake task:
namespace :graphql do
desc ‘Dump the graphql schema without comments into a file schema.graphql’
task schema_dump: [:environment] do
$stdout = File.new(‘schema.graphql’, ‘w’)
$stdout.sync = true puts GraphQL::Schema::Printer.print_schema(::Schema)
Store your schema output wherever best suits your needs (we are using AWS S3 for these contract tests):
s3 = Aws::S3::Client.new(YOUR_CREDENTIALS_HERE) File.open(‘schema.graphql’) do |file|
s3.put_object(
bucket: ‘contract-test-artifacts’,
key: “graphql_#{ENV.fetch(‘ENVIRONMENT’)}_schema.graphql”,
body: file
)
end File.delete(‘schema.graphql’)
end
end
As a part of your server build process, you will also want to check your schema against the client queries before deployment, but we’ll get to that later.
Check your queries on the client application build
Chances are you are already running a linter as a part of your CI build and if you’re running a modern JavaScript environment, eslint is an obvious choice. You can follow the instructions in the eslint-plugin-graphql repo for this part, but your config will look something like:
// In a file called .eslintrc.js
module.exports = {
parser: “babel-eslint”,
rules: {
“graphql/template-strings”: [‘error’, {
env: ‘literal’, schemaString: fs.readFileSync(“./schema.graphql”),
}]
},
plugins: [
‘graphql’
]
}
As a step of your build, pull down the schema from your cloud service and run the linter against it:
aws s3 cp s3://contract-test-artifacts/graphql_${ENVIRONMENT}_schema.graphql# or `npm run lint`
yarn run lint
Boom! We’re halfway there, and we’re already winning. You can now deploy your client applications safely, knowing all your static queries are valid.
Validate changes to your provider against your client application queries
Note: Your provider CI build will need to support all the dependencies of the eslint-plugin-graphql project. Fortunately, we already have node in our GraphQL provider project, so this didn’t add any meaningful overhead to our build process. This may not be ideal for everyone, however, and I offer some potential solutions and improvements to this strategy at the end of the article.
On each successful deployment of your client applications, upload all your .graphql projects to cloud storage. You may put each client application in its own directory in your bucket:
aws s3 cp ./ “s3://contract-test-artifacts/graphql_${ENVIRONMENT}_queries/my-client-application” — exclude “*” — include “*.graphql” — exclude “*schema.graphql” — recursiveBefore deploying your provider, pull down all your client queries. We again wrote a rake task for to pull the queries down and run our schema introspection:
desc ‘Load queries and schema JSON from contract testing’
task load_graphql_queries_and_schema: [:environment] do
puts ‘Writing schema.json…’
File.open(‘schema.json’, ‘w’) do |f|
f << Schema.execute(GraphQL::Introspection::INTROSPECTION_QUERY).to_json
end puts ‘Connecting to AWS S3…’
s3 = Aws::S3::Client.new(SECRET_STUFFS) contract_bucket = Aws::S3::Bucket.new(
‘contract-test-artifacts’,
client: s3
) puts ‘Downloading .graphql files…’
contract_bucket.objects.each do |obj|
if obj.key.match(/^graphql_#{ENV[‘ENVIRONMENT’]}_queries\/.*\.graphql$/)
dirname = File.dirname(obj.key) unless File.directory?(“#{Rails.root}/queries/#{dirname}”)
FileUtils.mkdir_p(“#{Rails.root}/queries/#{dirname}”)
end obj_data = obj.get.body File.open(“#{Rails.root}/queries/#{obj.key}”, ‘wb’) do |f|
f << obj_data.read
end
end
end
Run the above rake task and eslint to validate your schema against your clients:
set -eRAILS_ENV=development bundle exec rake db:create load_graphql_queries_and_schemaecho ‘Validating GraphQL schema has no breaking changes against consumer queries…’yarn && yarn run lint:contract
And that’s it! You’ve got a super-simple, damn speedy check that your changes won’t break any of your API consumers. Of course your apps need unit and functional tests to handle pesky details such as your code working correctly (did we mention that you can use that same schema.graphql file to mock your server for feature tests?). However, this is a big win for minimal effort. Of course, there’s always room for improvement…
Shortcomings and possible improvements
I briefly mentioned above that this potentially adds some dependencies to your provider and client applications. One future improvement we would like to implement is to create a service that runs these validations for us. Our clients would send up their queries and our backend would send up its schema and the contract service would be the one say “Yea” or “Nay.” Some possible extensions could be tracking which nodes in our schema are not being used and ask “Who’s using what?”
Additionally, your provider may mask certain fields from certain clients, so ideally, you’ll provide that version of the schema on a 1:1 basis with all registered clients. Again, the contract service could be helpful in this regard.
Wrapping up
Got any questions, comments or suggestions for improvement? Please let us know! Oh yeah, did I mention? PolicyGenius is hiring! If you love nifty technologies like GraphQL and you want to solve real world problems with passionate engineers, then check us out.
