EXPEDIA GROUP TECHNOLOGY — SOFTWARE
The Weird World of gRPC Tooling for Node.js, Part 1
How to build a JavaScript server you can live with
One would expect a slick, straightforward integration between Node.js and gRPC — they’re both Google’s progeny, after all. That’s sadly not the case. It’s complicated. At Vrbo™ (part of Expedia Group™), I’ve learned about this ecosystem and can help you understand the tooling landscape, learn how to use the tools, and see what kind of code you can expect from each. And steer clear of some pitfalls.
Key concepts
The major actors in this space have confusingly similar names and mismatched npm module names. Complicating matters further, the relationship between gRPC and Protocol Buffers makes it easy to conflate them.
- Protocol Buffers (a.k.a. protobuf) is a marshalling library that defines a particular way to represent data as bytes that can be saved to disk or sent over a network; it doesn’t care what you do with those bytes. It uses IDL files, typically with a
.proto
extension, to describe the structure of data in a language-neutral way. Protobuf provides tooling for several languages to turn the bytes into objects in those languages. This can be done either with static code generated at build time or with reflection-based library code at runtime. The protobuf schema language also includes abstract constructs for defining RPC services. - gRPC is a network message interchange specification that enables remote procedure calls between clients and servers using HTTP/2. The specification provides for synchronous, asynchronous, and streaming semantics. gRPC is also a suite of open-source tools and libraries for various languages that allow one to implement clients and servers that communicate using network messages. Just as with protobuf, there are solutions for implementing clients using static or dynamic code generation.
- The confusing part is that gRPC messages can be marshalled using any format, but the gRPC service interface contract is defined using
service
/rpc
keywords in.proto
files; the abstract idea of remote procedure calls is part of protobuf, independently of gRPC, and gRPC leverages it. In practice, gRPC uses protobuf marshalling by default and is the choice of least surprise in the community.
Cast of characters
Let’s introduce the suite of tools and libraries used for gRPC in JavaScript:
protoc
The core reference implementation of protobuf’s code generator for a variety of output languages, implemented in C++. You can use it to generate code to build and parse data in the protobuf format. If you dive into its js
module’s README, you’ll find this entirely accurate, four-year-old statement:
The API is not well-documented yet.
To hammer the point home, the tutorial covers several languages, but not JavaScript. This will be a recurring theme.
protobufjs
This is a third-party alternative to protoc
that is implemented entirely in JavaScript. It only handles protobuf, not gRPC. If you use protobufjs
on .proto
files that include service
and rpc
structures, those keywords will be ignored.
grpc (C library)
The core reference implementation of gRPC. This library provides the tooling and runtime libraries to implement the guts of gRPC. One does not use this directly unless one is writing a C or C++ client or server.
grpc (npm
package)
A library providing JavaScript bindings for the native grpc
. It uses node-gyp
to integrate the native runtime library into a Node.js API. This is the only widely used runtime for Node.js gRPC servers that I’ve found.
grpc-tools
gRPC expresses its service interfaces using keywords included in the protobuf schema language, namely service
and rpc
. grpc-tools
provides a plugin to protoc
that allows it to generate JavaScript client stubs and server skeletons that implement a normal object interface.
@grpc/grpc-js
In its sparse documentation, grpc-js
declares that it is:
- A pure Javascript gRPC client
- A beta-level release
To the first point, I recently came across this blog from Joyent describing recently-added server support in grpc-js
, and indeed, there it is. But given the second point, and the fact that there are no examples other than the tests included in the repo, I must conclude this isn’t relevant to production server applications.
grpc-js
leans on @grpc/proto-loader
to process the .proto
files.
@grpc/proto-loader
This is a wrapper over protobufjs
that provides concrete code generation for the gRPC constructs in a .proto
file. This only works for dynamic code generation, not static.
Two roads diverged in a wood
With apologies to Robert Frost, there are only two roads through the woods described above that lead to a working Node.js gRPC server on production-ready libraries:
- Static generation, using
grpc-tools
and thusprotoc
, plusgrpc
at runtime. - Dynamic generation, using
@grpc/proto-loader
and thusprotobufjs
, plusgrpc
at runtime.
Which you choose dramatically affects your development experience. In the next installment of this series, we’ll try static generation on a sample application. And in the final installment of the series, we’ll try out dynamic generation.