EXPEDIA GROUP TECHNOLOGY — SOFTWARE

The Weird World of gRPC Tooling for Node.js, Part 1

How to build a JavaScript server you can live with

Photo courtesy of Morgan Harris on Unsplash

One would expect a slick, straightforward integration between Node.js and gRPC — they’re both Google’s progeny, after all. That’s sadly not the case. It’s complicated. At Vrbo™ (part of Expedia Group™), I’ve learned about this ecosystem and can help you understand the tooling landscape, learn how to use the tools, and see what kind of code you can expect from each. And steer clear of some pitfalls.

Key concepts

The major actors in this space have confusingly similar names and mismatched npm module names. Complicating matters further, the relationship between gRPC and Protocol Buffers makes it easy to conflate them.

• Protocol Buffers (a.k.a. protobuf) is a marshalling library that defines a particular way to represent data as bytes that can be saved to disk or sent over a network; it doesn’t care what you do with those bytes. It uses IDL files, typically with a .proto extension, to describe the structure of data in a language-neutral way. Protobuf provides tooling for several languages to turn the bytes into objects in those languages. This can be done either with static code generated at build time or with reflection-based library code at runtime. The protobuf schema language also includes abstract constructs for defining RPC services.
• gRPC is a network message interchange specification that enables remote procedure calls between clients and servers using HTTP/2. The specification provides for synchronous, asynchronous, and streaming semantics. gRPC is also a suite of open-source tools and libraries for various languages that allow one to implement clients and servers that communicate using network messages. Just as with protobuf, there are solutions for implementing clients using static or dynamic code generation.
• The confusing part is that gRPC messages can be marshalled using any format, but the gRPC service interface contract is defined using service/rpc keywords in .proto files; the abstract idea of remote procedure calls is part of protobuf, independently of gRPC, and gRPC leverages it. In practice, gRPC uses protobuf marshalling by default and is the choice of least surprise in the community.

Cast of characters

Let’s introduce the suite of tools and libraries used for gRPC in JavaScript:

protoc

The core reference implementation of protobuf’s code generator for a variety of output languages, implemented in C++. You can use it to generate code to build and parse data in the protobuf format. If you dive into its js module’s README, you’ll find this entirely accurate, four-year-old statement:

The API is not well-documented yet.

To hammer the point home, the tutorial covers several languages, but not JavaScript. This will be a recurring theme.

protobufjs

This is a third-party alternative to protoc that is implemented entirely in JavaScript. It only handles protobuf, not gRPC. If you use protobufjs on .proto files that include service and rpc structures, those keywords will be ignored.

grpc (C library)

The core reference implementation of gRPC. This library provides the tooling and runtime libraries to implement the guts of gRPC. One does not use this directly unless one is writing a C or C++ client or server.

grpc (npm package)

A library providing JavaScript bindings for the native grpc. It uses node-gyp to integrate the native runtime library into a Node.js API. This is the only widely used runtime for Node.js gRPC servers that I’ve found.

grpc-tools

gRPC expresses its service interfaces using keywords included in the protobuf schema language, namely service and rpc. grpc-tools provides a plugin to protoc that allows it to generate JavaScript client stubs and server skeletons that implement a normal object interface.

@grpc/grpc-js

In its sparse documentation, grpc-js declares that it is:

• A pure Javascript gRPC client
• A beta-level release

To the first point, I recently came across this blog from Joyent describing recently-added server support in grpc-js, and indeed, there it is. But given the second point, and the fact that there are no examples other than the tests included in the repo, I must conclude this isn’t relevant to production server applications.

grpc-js leans on @grpc/proto-loader to process the .proto files.

@grpc/proto-loader

This is a wrapper over protobufjs that provides concrete code generation for the gRPC constructs in a .proto file. This only works for dynamic code generation, not static.

Photo courtesy of Oliver Roos on Unsplash

Two roads diverged in a wood

With apologies to Robert Frost, there are only two roads through the woods described above that lead to a working Node.js gRPC server on production-ready libraries:

• Static generation, using grpc-tools and thus protoc, plus grpc at runtime.
• Dynamic generation, using @grpc/proto-loader and thus protobufjs, plus grpc at runtime.

Which you choose dramatically affects your development experience. In the next installment of this series, we’ll try static generation on a sample application. And in the final installment of the series, we’ll try out dynamic generation.