EXPEDIA GROUP TECHNOLOGY — SOFTWARE

The Weird World of gRPC Tooling for Node.js, Part 2

Boldly building a statically-generated JavaScript server

Published in

Expedia Group Technology

6 min readJun 16, 2020

You can create a gRPC server in Node.js with static code generation, but with tradeoffs that you should understand. In the previous story in this series, I explained the ecosystem of gRPC tooling for Node.js. Now I’ll show you how to build a statically-generated server using those tools.

A carnival ride resembling a complicated orrery at Disneyland in Paris — Photo courtesy of Antonio Esteo on Pexels

A sample application

We’ll use the tools to build a simple sample application. This application provides a service that models a library filled with books. Its methods allow retrieving all of the books and checking out a book. All the code I’ll share can be found here.

The schema follows a structure recommended by Uber and enforced with their protobuf linter, prototool. Note that if you run the linter in my example code, prototool is run using a Docker image, so you’ll need Docker installed. The directory structure for the schema matches the package names used inside the schema, as recommended by Uber:

proto
├── com
│   └── rcbrown
│       └── grpc
│           └── v1
│               ├── library_api.proto
│               └── library_book.proto
├── package.json         # For linting scripts
└── prototool.yaml       # To configure rule exceptions

The schema itself is straightforward, and I won’t explain it much as I’m focusing on tooling. The use of request and response messages is another Uber recommendation.

Importantly, the example assumes that the data source underlying the service returns idiomatic JavaScript objects that match the protobuf schema. This would typically be the case when using NoSQL databases like Amazon DynamoDB or Mongo.

Static code generation with `protoc`

Use protoc to generate both the gRPC and protobuf artifacts using an npm script similar to this:

grpc_tools_node_protoc                                     \
    --proto_path=../../proto                               \
    --js_out=import_style=commonjs_strict,binary:generated \
    --grpc_out=generated                                   \
    ../../proto/com/rcbrown/grpc/v1/*.proto

If you don’t ask for commonjs, you’ll get Closure modules. If that’s your thing, give it a try because it has some additional features; I’m going with CommonJs for its ubiquity.

You can see it in situ in the package.json in my sample. It generates four files: two _pb.js files for marshalling objects, and two _grpc_pb.js files containing service metadata.

generated
└── com
    └── rcbrown
        └── grpc
            └── v1
                ├── library_api_grpc_pb.js
                ├── library_api_pb.js
                ├── library_book_grpc_pb.js
                └── library_book_pb.js

Here are some snippets from the generated library_api_grpc_pb.js:

So this really only deals with the service and rpc parts of the proto. Here are some snippets from library_api_pb.js:

That deals with all the marshalling and demarshalling for the request/response message statements defined in library_api.proto.

Similar to library_api_pb.js, library_book_pb.js contains marshalling code for the message statements that define the library book data structure itself in library_book.proto. It also contains an enum for its checkout status:

library_book_grpc_pb.js is empty because library_book.proto doesn’t contain any service or rpc statements.

Did you notice this?

library_api_grpc_pb.js:var com_rcbrown_grpc_v1_library_api_pb =
  require('../../../../com/rcbrown/grpc/v1/library_api_pb.js');
...
function serialize_com_rcbrown_grpc_v1_GetBooksRequest(arg) {
  if (!(arg instanceof
     com_rcbrown_grpc_v1_library_api_pb.GetBooksRequest)) {
...
----------------------------------------------------------------library_api_pb.js:proto.com.rcbrown.grpc.v1.GetBooksRequest = function(opt_data) {
...
goog.object.extend(exports, proto);  // Adds proto fields to export

The protobuf created a GetBooksRequest constructor that is deeply nested to match the package structure of the proto. But the gRPC layer expects GetBooksRequest to be at the very top of the exports. Though these came from the same code generation run, they don’t interoperate. If you build a server with these, you will be rewarded with 13 INTERNAL: Cannot read property ‘deserializeBinary’ of undefined when you call this service.

Punishing packages

In a dark corner of the protobuf documentation, Google admonishes that generated code that uses CommonJS imports (as opposed to Closure imports, the only other supported option) ignores package directives. That wasn’t exactly my experience. You can see above that protobuf did respect the package structure. But the generated *_grpc_pb.js files didn’t. They assume that protobuf objects won’t be in a package, so they refer to objects that won’t be there at runtime. The only solution is to not use packages with protoc and static code generation, which forces us to deviate from Uber’s recommendations. Put succinctly,

Packages don’t work!

For the rest of this installment in the series, I will use .proto files with no package directive and a flat directory structure:

proto
├── library_api.proto
├── library_book.proto
├── package.json
└── prototool.yaml

Writing the server

Pass the generated descriptor for the service and an object containing handler functions for its endpoints:

Each of the handler methods must transform the incoming messages from protobuf objects into regular JavaScript objects to pass to the underlying data access object (DAO), and reverse the process for return values. These transformations use the marshalling/demarshalling code generated in the _pb.js files.

Note that both handlers assume that the DAO returns a CheckoutStatus as a string that matches the names in the enum.

What could be simpler?

Java… Script?

I hope I triggered an eyeroll with that last sentence. If that looked simple to you, then you have lived a hard life.

Experienced JavaScript developers should bristle at the use of getters and setters. This is Java style, applied to JavaScript. This is particularly poignant because JavaScript has long had support for getters and setters that look like idiomatic field accesses, but the generated code didn’t employ them.

This sort of code doesn’t add value

But that’s just a detail — the architecture is more concerning. The handler code is very tightly coupled to both the protobuf data types and the native data types used by the DAO. Model changes will require detailed updates in the handler code, and if it’s overlooked, many sorts of changes will result in silent omissions.

If requests or responses contain complicated object graphs, this marshalling code in the handler will be quite involved. This isn’t evident from most of the examples you’ll find online (or from mine, really) because they usually use trivial messages. Similar API platforms in other ecosystems (such as Jersey for Java) automate this kind of work. (I’m not comparing gRPC to JavaScript frameworks like Express because they typically converse in JSON, which is native to JavaScript, sidestepping this issue.)

If you commit to using statically-generated Node.js gRPC code, you will have very detailed control over marshalling, whether or not you want it. In my mind, this sort of code doesn’t add value and I try to minimize it.

Writing the client

Writing a statically-generated client isn’t much different from writing a statically-generated server. Given that the client has the same .proto files available to it, and that the code generation has been done the same way as for the server, all you need to do is marshal requests, pass them to the generated client class, and unmarshal the responses. Here’s the complete client:

I needed to write mapCheckoutStatusToString to map the enum values from integers to strings; I didn’t need to do it for the server because the DAO automagically knew what strings to use.

Libraries are available for promisifying the calls, and some of the marshalling code could have been factored out, but I wanted to make the code as straightforward as possible and entirely visible. You can see that the marshalling forms the bulk of the code and is similar to the server.

Two roads diverging in a snowy forest — Photo courtesy of Oliver Roos on Unsplash

The road less traveled

You’ve seen some warts on writing a statically generated server.

Packages don’t work
You have to write lots of idiosyncratic marshalling code (though if your client or data layer model data much differently from the protobuf schema, you will be writing similar code anyway)

What you do get, however, is easy visibility into the code you must write. If you have to write code like this without the statically generated code,

const booksPb = getBooksResponsePb.getLibraryBooksList();

then you will have a very hard time knowing what you should type after the dot. After all, library_books_list isn’t in the .proto; it’s invented by the framework. It’s easy enough to open up library_api_grpc_pb.js and see what’s available.