GENERATING TYPE-DEFS FOR ENTERPRISE FEDERATED GRAPHQL: THE WEBILL WAY

PART 1: THE PROBLEM STATEMENT AND THE END SOLUTION

The entire WeBill development ethos is underwritten by the guiding principle that all of our developers must adhere to the WeBill design and methodology. When we carry out code reviews all the code developed by Tim must be indistinguishable in style and form from the code developed by myself or Jedd. In an enterprise scale development of the magnitude of WeBill single-man dependency and individual idiosyncrasies are anathema. New developer recruits are taught the methodology, design and style, and, within a month or two, can swim in the code developed months ago by developers who have long departed from the company. Sustainability of code is crucial and methodology is key. What I mean by “methodology” is difficult to articulate but it includes design, style, template and convention. We strive to be at the cutting edge of technology, and, at times, templating and convention can dumb down complexity, but it works for us. We have a factory-line development process which we believe deals with cutting edge technology and issues. Learn the method and the complexities of all new technology we adopt becomes trite.

Recently we decided to move our entire enterprise to the newly released Apollo GraphQL Federation. GraphQL Federation was the answer to the micro-service conundrum and multiple product nature of our business. We have a number of products which share types amongst themselves. Each product was supposed to be a discrete, independent micro-service environment of its own. Unfortunately, on the graphql level because types and constructs were shared, we had to break the Chinese walls we wanted to set up between each product. We could not get “stitching” to work for us. GraphQL federation was the answer.

The problem was getting GraphQL Federation on enterprise scale to work in our factory-line approach. This problem was exacerbated by our insistence on the WeBill “methodology”. The GraphQL Federation documentation was adequate but, as with all documentation and tutorials, did not give an answer to our enterprise concerns. In this series of articles I will walk through our methodology.

The following are the series of articles I plan to write on this topic:

1. Describe the problem and show the end result

2. Explain the user of functional programming and deployment of the ramda library to create the magic

3. Code walk-through of the utility library responsible for the solution

A: THE PROBLEM

At WeBill we have a number of inter-linked products. Field Manager is a web and mobile product for utility field workers who read meters, install meters, disconnect and connect meters and perform other field work on utility assets. Facility is a web and mobile product for facility managers to manage utilities in facilities such as shopping malls, office blocks, hotels, residential complexes and so on. A WeBill User can sign on and use both Field Manager and Facility products. The same User credentials are valid in both products. User management is handled in the Org product which deals inter alia with Users, Roles and Security.

All the above products, Field Manager, Facility and Org (there are many more) have their own data-bases, graphql servers and AWS assets such as dynamo-db, step functions and lambdas. These artefacts are not shared between products. Within each product there are a number of modules. Facility for example has over 20 modules such as Team, Instruction, Contractor and Device. Each Facility module has its own graphql types, inputs, queries, mutations and subscriptions. The Facility graphql artefacts are hosted in the Facility GraphQL Federated Service. We use a mono-repo and each product is a project within a lerna mono-repo. (I will impress upon our devops manager, Jedd, to write a series of articles on how we set up lerna).

Essentially a single project in most cases maps to a product. Each product project has the following packages:

• common — code shared by all the packages in the project
• gql — graphql definitions, resolvers and the Graphql Federated Server.
• lib — the domain logic
• sls — AWS lambda and step functions and other AWS code
• test — comprehensive unit tests for sls and lib.

Sample WeBill package project

This series of articles is going to focus on the gql package of the product project. As dictated by Apollo Federation, there is a separate project called gateway which federates all of the individual project GraphQL federated services.

Because there are so many modules in Facility, each module graphql types need to be defined in separate folders. If we wrote all of Facility graphql definitions on a single file, the file would be thousands of lines. Not only is it unwieldy and well-nigh impossible to work on such a massive graphql definition file, a single file also rules out more than one developer working on the graphql definitions at the same time. Recall Facility has a number of modules and more than one developer is working on a different module of Facility at any given time. Moreover, the tooling and linting for graphql is not all-together there yet, so when defining types at scale, it can take days to find the errant “:” in a single definition on the massive single file that is preventing your graphql types from being successfully parsed.

We therefore had no choice but to define the graphql types for each module within the Facility project separately from the other modules. In fact we took it to the extreme. Each module had to have its own file for graphql schema, inputs, query, mutation and subscriptions.

Each module has its own resolver. This series of article however will not cover with how we deal with resolvers.

Before federation, we managed to modularize our disparate graphql definition files more or less following the method described in the official Apollo GraphqlServer documentation for modularising resolvers with lodash’s merge.

The structure of a module before federation was as follows:

Previous gql project structure

The schemas were described in each file and the index.js of the schema folder merged the files. The typedefs.js merged all the modules into a single graphql type-def. What follows is a grossly simplified sample of the Contract module in the old Facility gql project.

contract.js

‘use strict’;
const { gql } = require(‘apollo-server’);
const contract = gql`
  type Contract {
  id: ID!
  contractId: String!
  description: String
  end: String
  mappingId: String
  start: String
  status: String
 }`;
module.exports = contract;

contract-input.js

‘use strict’;
const { gql } = require(‘apollo-server’);

const contractInput = gql`
  input ContractAddInput {
  contractId: String
  description: String!
}`;
module.exports = contractInput;

contract-queries.js

‘use strict’;
const { gql } = require(‘apollo-server’);
const contractQueries = gql`
extend type Query {
  contracts: [Contract]!
  contract(contractId: ID!): Contract!
  }`;
module.exports = contractQueries;

contract-mutations.js

‘use strict’;
const { gql } = require(‘apollo-server’);
const contractMutations = gql`
extend type Mutation {
  addContract(input: ContractAddInput!): Contract!
  updateContract(contractId: ID!, input: ContractUpdateInput!):Contract!   
}`;
module.exports = contractMutations;

index.js

This index file consolidates all the gql definitions for the module (here contract) within the project. Recall, there are numerous modules within each project.

‘use strict’;
module.exports = [
require(‘./schema’),
require(‘./mutation’),
require(‘./query’),
require(‘./input’),
];

typedefs.js

This file consolidates all the module index.js files into a single typedef schema to pass to the graphql server. There are normally over 50 modules per project; for purpose of this article I have simplified the typedef consolidation to only two modules for the project: billing and contract.

‘use strict’;
const billing = require(‘./billing’).schema;
const contract = require(‘./contract’).schema;

const typeDefs = […billing, …contract];

module.exports = typeDefs;

When we moved over to federation, this simple array-based technique used did not work. We could never get our Federated GraphQL Service to parse the schema, compile and launch. We tried a number of alternatives using lodash merge, ramda mergeAll and restructuring our module layout. In the end we decided to make a radical change to our original module design which had served us well for the past 18 months.

B: THE NEW STRUCTURE

The new module structure we finally adopted not only worked but also:

• Abstracted our graphql definitions from our code.
• Modularised our graphql files; a key goal we had set for ourselves.
• Untangled our resolvers from our graphql definitions.
• Removed the previous tedious typedef consolidation code per module.
• Grossly simplified the project typedef consolidation of all modules from hundreds of lines of code to a single line of code.

The new structure now looks like this:

gql project folder structure

All the type-definitions are in the gql folder; all the resolvers are in the modules folder, a single resolver per module. The folder sub-structure of the modules folder is exactly the same as the folder sub-structure of the gql folder. Convention is paramount and is the leitmotif of our new design.

Each module of the project has its own folder within both gql and modules. So the gql will have an agent folder and the modules will also have an agent folder. The resolver of the agent is defined in the modules folder. There are no graphql definitions in the modules folder.

modules folder for gql project

The graphql definitions for the agent module are in the agent folder of the gql folder. By convention the definition files are all of type .gql and there can only be the following types of file by strict convention:

• input.gql

All inputs and enums are defined here. By convention our input types all end with “Input”.

input AgentInput {
 cell: String!
 email: String
 firstName: String!
 lastName: String!
 idNumber: String
 photo: String
}

By convention all our enum definitions always begin with E.

enum EUtilityType {
GAS
ELECTRICITY
WATER
}

• mutation.gql

Because we are using federation, we need to extend the mutation definition. We do this for each and every mutation.gql in each module in the project.

extend type Mutation {
  “””
  Roles: Agent Admin
  Add a new Agent to an Org.
  “””
  createFacilityAgent(input: AgentInput!): Agent!
}

• query.gql

As with mutation, we extend Query in every single query.gql. WeBill insists that each mutation and query is documented in the definition file. Documentation consists of enumerating the roles that can execute the query or mutation and a one-liner explaining the query or mutation.

extend type Query {
 “””
 Roles: Agent Admin, Agent User
 Retrieve all Agents for the Organization (enabled and disabled)
 “””
 facilityAgents: [Agent]!
}

• schema.gql

All the type schemas for the module are defined here. The following type definition can be exported to other federated graphql servers.

type Agent @key(fields: “id”) {
id: ID!
cell: String!
email: String
firstName: String!
idNumber: String
lastName: String!
photo: String
status: String!
}

The overall file structure of the agent module gql module of the Facility project looks like this.

agent module gql def files

We create the entire Facility project consolidated typedefs with a single one liner as follows. This code is in the index.js of the modules folder:

‘use strict’;
const { makeTypeDefs } = require(‘@wb/wb-server-common’).lib.gql;
const typeDefs = makeTypeDefs(__dirname, ‘../gql’);
module.exports = typeDefs;

As far as the author of the graphql federated service is concerned, all he/she has to do is author his/her graphql file definitions in each module for each project. Then to create a single typedef to pass to the federated graphql server, the above single line code snippet will do the trick.

What this code does is traverse the entire gql folder of the project module by module (each module with its folders and input, schema, mutation and query files) and consolidates a single parsed typedef to pass to the federated graphql-server. Clearly, all the magic happens in the makeTypeDefs method of the common project. The common project is referenced by every single project in our mono-repo. All the common utilities, libs and configs are found here.

The next article in this series will explain the technology deployed in makeTypeDefs with particular emphasis on our use of ramda. The final article will walk through the end to end code of makeTypeDefs with emphasis on how we structure our utility code files at WeBill. At the end of the series we will be open-sourcing our solution on github with a sample project.