Project: OpenAPI definition generator

This article is also published on my website: https://fwouts.com/articles/project-openapi-definition-generator/

This is a first attempt at documenting the objective, background and design of Zenc Labs projects before actually building them. I’m trying this for a few reasons:

1. It will give visibility to fellow developers on what I’m building, and hopefully they’ll give me useful feedback and let me know about similar/related projects.
2. It will help me “freeze” the medium/long-term vision of the project before getting my hands dirty. It’s too easy to forget your original goals and lose motivation when you’ve been working on a project for a few weeks. It never feels “complete”, and it is notoriously impossible for a human to remember exactly how they thought about a problem in the past. Unless they’ve got it written down.
3. Last but not least, documenting projects publicly will help me stay accountable, get them implemented and go through the extra work of publishing them.

Enough said, onto the actual project details!

Background

OpenAPI, formerly known as Swagger, is becoming an industry standard to document HTTP-based RESTful APIs.

OpenAPI logo

Unless you use GraphQL or gRPC, your API can probably be described in detail with a YAML file following the official OpenAPI specification. Whether your endpoints handle JSON, XML or multipart forms, you can document their inputs, outputs and status codes with an OpenAPI definition. It can then be used to write tests, generate client libraries or run server stubs.

Here’s an example from the official OpenAPI 3.0 repository:

The OpenAPI format can be a bit intimidating for someone who isn’t familiar with its structure. Adding a new endpoint, defining a new type or editing an existing one can be tricky. The API definition can also get quite large, and splitting it into multiple files requires workarounds.

Developers can use tools like Swagger Editor to edit an API definition more easily. Such tools are essential to ensure that the produced OpenAPI definition is valid. However when editing the OpenAPI definition in a code editor, you’re on your own. It’s just YAML.

As of June 2018, a lot of tools in the OpenAPI ecosystem only support OpenAPI 2.0 (aka Swagger 2.0) definitions. Writing an OpenAPI 3.0 definition means that you’ll have limited tools for a while. For example, swagger-codegen isn’t yet stable with OpenAPI 3.0.

Objective

We’ll create a higher-level DSL to generate OpenAPI definitions which:

• is easier to learn
• is language-agnostic (independent of server language/framework)
• can be split into multiple files easily
• can output both OpenAPI 2.0 and 3.0 definitions on-demand
• validates itself

In the interest of efficiency, and to avoid expanding unnecessary effort without first validating the tool’s value in practice, we’ll first limit support to a subset of OpenAPI: JSON-based RESTful APIs.

Overview

We’ll create our DSL with Ruby.

This doesn’t mean that you need to use Ruby on Rails for your server. Your API definition is independent from your server implementation. Whether you use Java, Golang, Node.js, Rails or even C++, you can still use this DSL to generate your API definition.

Here’s what a simple API definition could look like:

Example API definition using our Ruby DSL

You generate the OpenAPI definitions with:

$ gem install apigen
$ ruby example.rb

Detailed design

Language

We need to pick a language to implement our DSL. Ruby makes it easy to do just that, and it has a minimalistic syntax that shouldn’t scare away too many developers. See Alternatives below for an in-depth of discussion of possible substitutes.

Functionality

Our DSL will initially provide the minimal amount of functionality to describe a JSON API.

In particular, you start by declaring an API:

You can then define as many endpoints as you’d like:

Each endpoint must also declare its input and output shape:

You’ll notice that we haven’t defined the type :user. Here’s how to add it:

Then, all that’s left is to validate the API and generate the OpenAPI definition:

API validation

Given the above structure, validating the API is a relatively straightforward process. We need to ensure that:

• Each endpoint has a valid input and output type
• Each path parameter (e.g. /users/{id}) is defined with a valid type
• Composite types (object and list) reference valid types

Generating the OpenAPI definition

To generate the OpenAPI definition, we need to create a hash (dictionary) and simply output it with the YAML module.

We’ll make sure that required fields such as contact can be set through the DSL. If they aren’t explicitly specified, we’ll show a warning and set them to a reasonable default (e.g. contact may be set to apigen@localhost).

Splitting up into multiple files

If you’d like to split up your API definition into multiple files, the shortest solution is to make api a global variable by prepending a $, then using Ruby’s require keyword:

Alternatives considered

Alternative languages

Although writing an API definition with our DSL only requires rudimentary understanding of Ruby, it still may be a different language from what developers are used to.

JavaScript is another candidate, since 70% of developers are familiar with the language. The only hiccup: it’s not as concise as Ruby and would require a package.json and node_modules to work.

Here’s what it could look like in JS:

Another possibility is a made-up language. I actually built this before learning about Swagger and OpenAPI (here). Here’s an example:

endpoint createUser: POST /users CreateUserRequest
-> success 200 CreateUserResponse
-> failure 400 string

type CreateUserRequest = {
  name: string
  password: string
}

type CreateUserResponse = {
  id: string
} 

There are a couple of issues with this approach:

• developers need to learn a new syntax
• code editors will lack syntax highlighting, making errors harder to catch
• it becomes a purely descriptive syntax (no way to add any dynamic logic)
• splitting an API into multiple files requires introducing a module/import system (possibly reinventing the wheel)

Future plans

Headers, authorisation and other formats

While the DSL described above is fairly limited, it should easily be extended to cover other concepts that OpenAPI provides, while keeping required effort from the developer at a minimal level.

Typing

Ruby may soon have typing, thanks to Stripe’s efforts. If code editor support follows, this may help provide helpful autocomplete suggestions to developers as they write their API definitions in Ruby.

Extension to non-REST APIs

The DSL could be extended to support non-RESTful HTTP endpoints, such as GraphQL schemas.

The same approach to API definition could also be used to promote modular code. You could describe the API of a local code module in an effort to split up a monolithic codebase into several components communicating locally, potentially via inter-process communication between different languages.

Implementation progress

Follow along at https://github.com/zenclabs/apigen. Feel free to send any feedback/ideas via Twitter or by email.

[ ✔ ] Reserve a Ruby gem (apigen) - 29 May 2018
[ ✔ ] DSL proof of concept in Ruby - 29 May 2018
[ ✔ ] Write a design doc - 5 June 2018
[ ✔ ] API validation - 8 June 2018
[ ✔ ] Unit testing - 8 June 2018
[ ✔ ] OpenAPI v2 generation - 9 June 2018
[ ✔ ] OpenAPI v3 generation - 9 June 2018
[ ✔ ] JSON Schema generation - 16 June 2018