Using protocol buffers with Node.js + Swagger + Angular

Recently, I started experimenting with Angular/Typescript and Node.js in a Google Cloud App Engine (GAE) flexible environment and I wrote a small app called Circly, you can find the source code in my github repository (still work in progress!).

The technology stack is quite complex and composed by:

Frontend development: Angular/Typescript + Material Design + Bootstrap UI components + AngularFire2

Frontend hosting / authentication: Firebase hosting with Firebase authentication using Google account

Backend RESTful APIs definition: Swagger / OpenAPI

Backend implementation: Node.js + Express

Backend hosting: Google App Engine Flexible Environment for Node.js

RESTful APIs delivery: Google Cloud Endpoints with api-key and JWT token security

CI and deployment: GitHub/Travis-CI.org

all written with ❤ in Atom

Try the app demo (works best on smartphones!)

This article is the first of a series of articles based on on the code I wrote for the Circly webapp.


Introduction to Protocol Buffers

One of the thing I invested some time, has been using Google protocol buffers (PBs) to share data between the backend APIs and the frontend and viceversa. If you don’t know what the PBs are, check out this definition from their official website:

Protocol buffers are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data — think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.

Basically, PBs are a way of encoding structured data in an efficient yet extensible format. Most of the existing webapps use JSON as mechanism to exchange data between frontend and backend. Protocol Buffers have several advantages over pure JSON:

  • it is very easy to describe the data using PB language-neutral syntax; this is an example of protocol buffer:
message Person {
required int32 id = 1;
required string name = 2;
optional string email = 3;
}
  • the schemas are automatically converted into modules that can be used directly in your code and PBs support several languages out-of-the-box, such as Go, Python, etc.

For instance, Circly frontend is written in typescript, while the backend is written in javascript (Node.js), so using PBs I defined the data schema once and generated the javascript prototypes and the corresponding typescript declaration automatically.

  • PBs can be used to describe both the messages exchanged by the services and the data stored in the backend databases.

Circly APIs are built with a CRUD model in mind, so most often the messages exchanged between frontend and backend represent entities stored in the database.

  • No need for validators, parsers, etc. Once a protocol buffer is specified with its schema, we can use pre-existing methods to parse, generate, and validate the instances (messages) described by that protocol buffer.

As you see, there are several good reasons to prefer PBs over pure JSON in our webapps. The reason why I keep saying “pure JSON” is because Protocol Buffers can be used in 2 different flavors:

  1. without any conversion to JSON format: this means that the backend and the frontend will only exchange pure binary data (the PB representation). Using this mechanism we exploit most of the advantages of PBs, including the fact that PB entities are usually smaller in size w.r.t. JSON objects.
  2. converting the PBs into JSON before sending it over the HTTP channel. In this situation, the PBs are converted into JSON format before the delivery and converted back into PB objects once they reach destination. With this mechanism, we still have the flexibility of protocol buffers (without the data compression advantages) but we can use them with systems that don’t play well with them.

Given that while building Circly, I already decided that I wanted to use Swagger/OpenAPI which works better with JSON, I decided to go for option 2 (anyway, I am planning to study a mechanism to use option 1 with Swagger/OpenAPI in future).

Let’s see step by step how to implement it in a web application, taking Circly as example.


Setup the basic frontend/backend structure

This guide assumes you already created a simple Angular application using Angular CLI (see the official documentation) and a basic backend API service using Swagger-node (the official GitHub page provides a lot of info on how to get started).

In my case I created the backend and the frontend in two different folders in the same GitHub project (frontend and backend).


Define the protocol buffer schema

Create a folder protos/ inside the backend folder and add a new file. In my case, I called it model.proto:

The syntax should be familiar to everyone with coding experience, if you have any doubt, please refer to the official documentation.


Generate the javascript prototypes and typescript declaration

Now, you can use the tools provided by the Protocol Buffer environment to automatically generate the object prototypes that you will use in your javascript code — for the Swagger + Express + Node.js backend — together with the typescript declaration file that you need to use the same prototypes as classes in your typescript code — for the Angular fronted. If you don’t know what is a declaration file, check out the documentation.

First of all, you will need to install protobuf.js. From the official website:

protobuf.js is a pure JavaScript implementation with TypeScript support for node.js and the browser.

Install the protobuf.js npm in the backend folder:

npm install protobufjs [--save --save-prefix=~]

At this point you can run this command to convert your .proto files into standard CommonJs modules that you can import in your Node.js backend:

node_modules/protobufjs/cli/bin/pbjs -t static-module -w commonjs -o model.js protos/model.proto

This will generate model.js and in the next section I will show how to use it in your code.

To generate the typescript declaration file you can run this command instead:

node_modules/protobufjs/cli/bin/pbts -o model.d.ts model.js

Notice that the input of this last command is the model.js file generated in the previous step and not the original model.proto. Indeed, you will need both files in your frontend app, as model.d.ts is just telling the typescript compiler which types are associated to the entities defined in the model.js.

You can see the files generated in output for Circly in this folder of the project GitHub repository.

To automate the process I created a quick shell script to execute these two commands and copy the two generated file also in the frontend folder — we will need them there later on. I placed the shell script directly inside the protos/ folder in the backend:


Generate the YAML specification

Now we need to configure Swagger so that our backend APIs use the same schema defined by the protocol buffers to exchange data with the frontend.

Swagger is configured using a YAML configuration file. This is a simple configuration:

As you can see our backend APIs:

  • expect in input JSON data and produce JSON output (consumes/produces section in the configuration above).
  • support only one path /api/v1/collection with POST/GET requests:
    - POST requests are mapped to AddCollection operations
    - GET requests are mapped to GetCollections operations
  • the configuration also describes the schema of the input/output data using the schema syntax:
schema: 
type: array
items:
$ref: “#/definitions/model.Collection”

The $ref symbol is telling Swagger to go and find the schema of the items in the array in another section of the configuration, called definitions. We haven’t defined this section yet (so this configuration won’t work at the moment) because we need to convert our protocol buffer schema in model.proto into the YAML format that can be used in this configuration.

To do that we can use a tool called protobuf-jsonschema which produces YAML and/or JSON schemas starting from a protocol buffer.

As usual install the NPM in the backend folder:

npm install protobuf-jsonschema -g

After that, you can use this command to convert the model.proto into a set of YAML definitions:

protobuf-jsonschema — format yaml protos/model.proto > model.yaml

The model.yaml will contain the YAML definitions that we have to copy in the Swagger configuration file created above. Let’s change it to look like this:

As you can see, I added a new section called definitions and copied the content produced by the tool inside there. Be careful to respect naming across the openapi.yaml file, i.e. if your $ref points to model.Collection then in definitions you must have a section called model.Collection or it won’t work, so you may have to do small manual changes to the generated file (I am planning to write a small shell script to automate all the steps).


Use the generated code in the backend

At this point, we just need to implement one of the operations described in the Swagger configuration so that the server will respond to the API requests with some data.

In this example I will implement only the GetAllCollections operationId corresponding to the GET requests to the api/v1/collection path. It means this function is called when the server receives a GET request to the aforementioned path.

If you created the project using the swagger command line tool, you just need to create a new js file with the same name of the x-swagger-router-controller (GetAllCollections.js) inside the api/controllers folder that should have been automatically generated for you. Here is the file for my Circly app:

The function GetAllCollections (notice that the name is the same as the operationId) is automatically called by the server: the req argument contain the request parameters, the res instead should be filled with the data we want to return back from the server.

First, you need to import both the protobufjs module we installed in the previous step and the module we generated before from our model.proto.

var protobuf = require(“protobufjs”)
var protos = require(“../../protos/out/ts/model.js”)

At this point, we are ready to create new protocol buffer objects in our code using the methods automatically generated for us by the framework:

...
// Creates a new model.Collection object.
var mock1 = protos.model.Collection.create()
// Sets the object fields values.
mock1.name = “mock collection”
mock1.createdTsMicros = 123456
mock1.description = “simple mock collection”
....

Pro tip: To simplify even more writing the code and if you use Atom as me, I suggest installing the atom-ternjs plugin: the plugin analyzes the model.js module and provides auto-complete support to create protocol buffer objects in javascript!

Cool, now we only need to return the array of PBs objects as JSON data, to do that we simply use the following command:

res.json([mock1, mock2])

Swagger will automatically parse these JSON objects and check they match with the YAML definitions we created in the previous step. This is an extra-check, particularly useful for parsing the input data coming to our APIs.


Use the generated code in the frontend

The last step is to use the protocol buffers in our Angular frontend with typescript.

Install the protobuf.js npm in the frontend folder as well:

npm install protobufjs [--save --save-prefix=~]

and copy the model.js and model.d.ts files generated before in the frontend as well (this is not mandatory, but I like to keep frontend and backend completely separated).

Now you can use the Angular HttpClient to send GET requests to the backend APIs and parse the responses:

Thanks to protocol buffers we can automatically type-check all the data coming from the server and create model.Collection objects from the responses sent by the server using auto-generated methods:

...
...model.Collection.fromObject(data)
...

…and Atom will automatically support auto-complete!

Moreover, this works two-ways: we can use protocol buffer messages to send type-checked requests to the server.

Neat!


Conclusions

I hope you are now ready to go to use protocol buffers in your web applications. There are many improvements to what I discussed here:

  • most of the steps can be easily automated, this is especially important to support continuous integration and deployment.
  • there is no need to convert .proto files into CommonJs modules as protobufjs can directly read from proto files saving us a conversion step
  • protocol buffers can be directly exchange between backend and frontend as binary data, avoiding the JSON conversion. It remains to be seen how Swagger can support this pattern.

As usually, please comment if you find any typo, improvement or error and feel free to reach me out directly.

Happy coding!