Bridging the Network Divide
Have you ever wondered why we build and support rich type and model information at our API and database layers only to throw that all away when crossing the HTTP divide to web and native clients? We on the grocery team at Walmart Labs are interested in many of the ideas that GraphQL, Falcor, and similar tools bring to the table. GraphQL’s type system that exists on both client and server and Falcor’s transparent model access over the network both enable very powerful and accessible ways of building and maintaining products.
Like most teams, we have a plethora of services already in production in addition to other constraints that slow or prevent the adoption of these specific tools. Though given the tools we have in hand we can reach and attain some of the promise these other abstractions provide.
While I am going to write about our specific implementation the idea I hope you capture is that you should not use your current circumstances or technologies to keep you from doing better today. There are likely ways to create more expressive abstractions or tools to give you more safety and stability to develop and deploy with confidence.
Specifically, my team has Swagger documentation generated from the API implementation itself. We can consume this information and feed it into JavaScript code generation tools like Babel and jscodeshift to generate our client side code. With that generated code annotated with Flow types we have quickly built a solid bridge between UI and server models.
A key benefit of doing this with a code generation model is we can generate on demand git diff’s to see all API contract changes made. Flow’s static analysis can subsequently guide us through any breaking changes. This gives us confidence that that our client code interacts with the API in safe and correct ways and provides convenient developer tooling for editor autocompletion when using methods that interact with the API.
Most modern server frameworks have tools to richly describe, validate, and share domain models. This often comes in the form of an ORM that abstracts the database operations from the application operations while providing a rich API to interact with these objects.
Rails has ActiveRecord.
Hapi has Joi validations.
GraphQL has its type system.
These abstractions make it trivial for server engineers to generate documentation and expose it to consumers (like UI engineers!).
But why stop there? Why throw away the tooling integration and safety of an API contract for some meager documentation at the network divide?
For the rest of this article I want to outline our current solution and end with some problems and next steps. Our approach has prioritized tackling the following two problems:
- How can we make API interactions predictable and automatic?
- How can we enable static analysis of UI code to the API request and response contracts?
We do not claim to be novel in our solution here. Prior art tackling similar problems are:
- Facebook’s GraphQL/Relay
- Google’s Protocol Buffers
- Netflix’s Falcor
Considering the trade-offs our solution sits between ad-hoc network calls and the above solutions. These solutions require an agreement between the client and server on a system beyond HTTP. Our solution allows us to take advantage of the existing tooling and infrastructure while adopting an incremental path towards more opinionated and powerful abstractions.
The Tools
- Swagger generates rich documentation from the implementation.
- jscodeshift to transform the Swagger representation into a JavaScript abstract syntax tree (AST) and source file.
We’ll use IBM Watson’s public Swagger definition. You can see the HTML view of this at https://watson-api-explorer.mybluemix.net/apis/text-to-speech-v1
The secret is that Swagger generates a JSON view of this as well. https://watson-api-explorer.mybluemix.net/listings/text-to-speech-v1.json
As programmers we’re used to working with data structures like objects and arrays and transforming them.
…if only we could transform Swagger’s JSON format to a JavaScript file…
Let me introduce you to the JavaScript Abstract Syntax Tree! Alas, don’t be frightened. Trees create the oxygen we breath!
An overly brief introduction to ASTs
A typical program is written and stored in a data structure known as a string. Strings aren’t a very rich data structure so most tooling converts the string into a rich layer called an abstract syntax tree — an AST.
A tree structure is just a collection of nodes. In JavaScript there are nodes for things like Identifiers, VariableDeclarations, FunctionExpressions, ReturnStatements and every other construct of the language. As a short example you may look at the following hello world function and its AST representation.
function hello () {
return “world!”;
}The function above can be represented as the following AST structure:
{
"type": "FunctionDeclaration",
"id": {
"type": "Identifier",
"name": "hello"
},
"params”: [],
"body": {
"type": "BlockStatement",
"body": [
{
"type": "ReturnStatement",
"argument": {
"type": "Literal",
"value": "world!",
"raw": "\"world!\""
}
}
]
}
}Putting the pieces together
We’ve taken a peek at the Swagger JSON model as well as what the source code of a JS program looks like. Now to write a program that converts the Swagger model to our JavaScript AST and print that into a file.
Let’s start by simply generating the export functions and signatures of the file. For each Swagger endpoint we want to generate a block of code that looks like:
type CustomType = any;
export function getV1Resource(
arg1 : string,
arg2 : string,
arg3 : CustomType
) {}It’s only about 130 lines of code to generate this for the entirety of the Swagger definition. That’s 130 lines of commented code including requesting and parsing the JSON itself.
https://gist.github.com/iamdustan/2e36a440b5702e14376bfba56d303427
Read through it. It may feel a bit foreign at first but it’s easy. Promise!
The heart of it is really this small:
/**
* Generate a JS AST from a Swagger entry.
*/
const toFn = (path, method, obj) => Object.assign(
j.exportDeclaration(false, j.functionDeclaration(
j.identifier(genFnName(method, path)),
genArgs(obj.parameters),
j.blockStatement([])
)),
{comments: [j.commentBlock(formatDescription(obj.description))]}
);We are avid users of Redux at Walmart Labs so next we’ll make the return value of the exported function declaration a simple object definition.
Let’s end this tour by writing the code to generate the following function body:
return {
type: 'FETCH',
payload: {}
};This is a ReturnStatement who’s value is an ObjectExpression with two keys.
const genBody = (path, method, obj) =>
j.blockStatement([j.returnStatement(
j.objectExpression([
j.property(‘init’, j.identifier(‘type’), j.literal(‘FETCH’)),
j.property(‘init’, j.identifier(‘payload’), j.objectExpression([]))
])
)]);Future challenges
At this point we have some large wins. We have rich type definitions for each model that the API may return. We can use these throughout our code base and whenever they change Flow will tell us what parts of our UI are performing invalid property lookups or missing a required input param to the API.
When we want to interact with the API we no longer need to leave our editor to see the API contract but can get that information directly from Flow.
Alas our type definitions for the return values are quite lossy. For example, if the acceptable input is {a: string} | {b: SomeEnum} what is currently being generated is {a: string, b: SomeEnum}which is entirely different. We are currently manually patching these cases. Now that our API has stabilized we’ve considered abandoning the auto-generated types for manually maintained definitions.
Lastly, we are considering exploring parameterized type definitions for our fetch middleware with the goal of helping Flow understand the precise shapes our Redux reducers need to account for.
Now if I can only find a way to get Rust’s exhaustive pattern matching
enforcement in JavaScript…
Thanks to Felix Kling and Joe Hudson for review and feedback on this article.
