How JavaScript works: Implementation of gRPC in a Nodejs application

Published in

SessionStack Blog

14 min readApr 21, 2022

This is post # 60 of the series, dedicated to exploring JavaScript and its building components. In the process of identifying and describing the core elements, we also share some rules of thumb we use when building SessionStack, a JavaScript tool for developers to identify, visualize, and reproduce web app bugs through pixel-perfect session replay.

Before we get to gRPC, we had REST, the most popular and most familiar architectural style for building APIs which are used for Web applications and connecting microservices. It also provides a request and response model of communication using the HTTP 1.1 protocol(this allows multiplexing). gRPC on the other hand works with a client response model of communication but uses the HTTP 2 protocol.

There is a lot to demystify before we face the implementation of gRPC in Nodejs, especially the mode of communication and even why, or better still what gRPC has to offer that REST does not.

In this article, our main focus is the implementation of gRPC in Nodejs, and we will learn when to use gRPC in your project and its Pros and Cons.

Prerequisites

Below are the requirements to fully understand and implement gRPC in Nodejs.

1. A Basic understanding of Nodejs

2. A basic understanding of building APIs

3. Nodejs 12 or above should be installed.

The Concept of gRPC

One of the hardest things about software applications these days is building the APIs, there is a lot to think about and even worry about. We need to think about the endpoints and if they should follow the REST standards, the data model of the API(JSON, XML, etc), how we can handle errors with these endpoints, and even how we could scale these endpoints to handle a thousand requests. Building an API should be more about the data than the above. And this is the where gRPC framework comes in. The Client makes a request to the server, and the data is sent to the Client as a response.

Before we discuss more, gRPC is a free and open-source Remote Procedure Call(RPC) framework developed by Google and can run in any environment. It is built on HTTP/2 which supports streaming. It is language-independent, has low latency, and makes client communication easier because you don’t have to focus on implementing the HTTP server, HTTP client, thread and asynchronous model and all of that. Rather the focus is on the business logic because the gRPC framework handles these for you. It uses Protobuf over JSON and XML which is a binary format. And this is much better and faster.

Remote Procedure Calls themselves are widely used in distributed systems where a client system communicates to a server, and this happens just by invoking a function on the server. The functions, in this case, are the business logic and the client only has to call those functions. There are other RPC frameworks aside from gRPC like COBRA or Apache Thrift but gRPC is loved because of its support for HTTTP/2. According to the official documentation, it also has pluggable support for authentication, load balancing, tracing and health-checking.

In gRPC, the server has methods that perform the business logic, and act like a service and these methods can be called by a client application. On the other hand, the client(people call it stub too) creates a channel that will be used to make remote procedure calls to the service. The diagram below which is from the gRPC site is a clear illustration of the Client and the Service.

Also, gRPC uses Protocol Buffers as its data format, which is serialized and structures the data. So, if a Client and Service want to communicate, Protocol Buffer is used to serializing the data and the data is lightweight and simpler to read than other formats like XML or JSON. A basic structure looks like this:

Each field contains the datatype(string), either required, repeated or optional and an identifier(= 1). This will be done in a .proto file. The protobuf compiler then generates a file for us to use in our application. We can use the file as seen below:

We will walk through the process during the Implementation

Methods of communication

Let’s talk about the methods or ways gRPC uses to communicate or a better way to say this is the gRPC life cycle. What happens when a gRPC client makes a call to a gRPC server? As we have seen, gRPC uses the Protocol buffers by default as its Interface Definition Language (IDL) to make a service interface and also define the structure of the payload messages. The underlying technology, RPC has a life cycle where each service has its method of communication. Below are the lists:

Unary RPC

Here, a client makes a request to the server and waits synchronously for a response. The server on the other hand does what it has to do; database query or computation or a call to another service.

Server streaming RPC

Another method of communication which a client makes a request to the server. Just like Unary but here the client is expecting a lot of data to be returned which is a stream. Multiple responses are expected to be returned. An example of this Server streaming is Netflix or YouTube videos.

Client streaming RPC

The client here sends a stream of messages or data continuously to the server. The server waits for the client to send all messages before it acknowledges and sends a response. For example, uploading a large file to the server will better be implemented using a stream to send chunks of the messages to the particular service.

Bidirectional streaming RPC

A combination of server and client streaming where both the server and client can send a stream of messages to each other. Both the client and server communicate independently. And the client can asynchronously send a stream of messages while the server is sending. An example of this is chatting, gaming, and so on.

gRPC vs REST vs GraphQL

As we know, these three API technologies are used as a data exchange layer where there is a communication between two components; backend and frontend or a backend and a backend communication. Knowing when to use one of these technologies has been a problem. If we look closely, we understand that some of them were rolled out to fix issues of others, for example, gRPC was designed to handle distributed and complex systems where we have data flowing around. This too works with REST but is not as efficient.

REST uses four standardized HTTP verbs to define interactions in its requests. These HTTP verbs(POST, GET, PUT, DELETE) are used to perform CRUD operations on resources. REST provides an endpoint that identifies the resource we want to query or mutate (/users/:id). REST has been in use since 2000, and it has gone matured over the years. And this is why a lot of people use REST to build API. Also, REST has a good convention.

GraphQL on the other hand lets you define your API using a schema. GraphQL Types, Queries, and Mutations are specified and the client can use the definition of those Queries and Mutations to create and query data. GraphQL focuses on delivering the exact data that is required rather than over fetching which is solving the problem of REST. Also, GraphQLis very flexible in any development environment.

In gRPC, the communication is done using a binary format called Protocol buffers. The Protocol buffers define our contract(methods) and payload messages and both have to be defined before doing anything. This is almost like REST but the data format is not in JSON. Also, this is more performant since the data format is binary.

Looking at the three, you will find out that none is much better than the other. One is only better than the other based on your requirements and what you want to achieve. In summary, REST works well if you are building for consumers and is easier to implement. It shines if you do not have an idea or are not sure of what your audience needs giving you the flexibility in the number of endpoints you can create. All these come with drawbacks but they do not matter since our consumers are unknown. It also supports caching easily so we do not have to make round trips to the resource storage for retrieval.
GraphQL works better when your client will be performing a lot of queries to the server and wants to avoid over fetching. It also works better when you have multiple clients making use of the data in various ways, for example, a web and mobile platform will not display the same data in the same view. Another use case is if you have multiple data sources, for example, you have a database, REST endpoint and JSON. GraphQL can help organize them into a single interface where the client has no idea they are of different data sources.

gRPC works well with distributed systems where you have backend servers talking to each other and the data exchange is much faster because of its binary format. gRPC is best for PolyGlot Microservices, where you have not standardized your runtime. Having a single protocol buffer file that would code-gen the client to the server and keep them in sync is a very powerful pattern.

When to use gRPC

As I described earlier, using gRPC works well when you have a server-server environment where one server invokes a call on another server. It is industrial intent meaning that it is specifically for machines to communicate and it’s not for the browser. It was designed to be fast and to stream data between nodes in a distributed environment. It is also built on HTTP/2 which allows a bidirectional flow.

Implementation

We are finally in this part of the article you longed for. With our understanding of the concept of gRPC, we can begin implementation. Our application will be a simple book store application where you can add books to the store and read books from the store. We will be building both the client and server with JavaScript.

What we will be implementing:

Protocol buffers
createBook service
readBook service
readBooks service

The first thing we need to do is to create a protocol buffer that is going to hold the interface/schema of our services and also the payload messages. This is important when we have a client and service system that is going to communicate.

Step 1: Create a protocol buffer file

To begin, create a folder and initialize your Nodejs project with the `npm init -y` command. This folder will contain all necessary files.

Create a folder called proto and that folder will contain your proto files. For now, we are needing just one proto file which is `bookStore.proto`:

Our file will need us to provide the syntax which is proto3 the version. After then we define the package name which we named bookStorePackage. In this package, we define just one service `Book` with three methods that will be invoked by a client. Note that we can also define more than one service in this file.

The createBook method takes in a parameter `BookItem` which we also defined to have the id and book. The response is also BookItem. The third method readBooks does not need a parameter to take in so we define it as empty or void. The response is an array of book lists.

Step 2: Create the server

Install these two packages before you begin:

Next, in the project directory, create a server.js file that will hold our service and its methods.
Now, add the code below to the server.js file:

In the above code, we imported two packages which we installed@grpc/grpc-js and @grpc/proto-loader earlier. The proto-loader loads the proto file for us, and this package compiles it to be flexible in any language. With that, we load the package defined which returns an object containing our package name.

Using grpc to create a new server, we bind our port `50051` to any port as well but I used `0.0.0.0`. The second parameter is an insecure server credential. We could provide a secure connection if you want to.

With the server created, the .addService method is called to add our defined services to the server. When done, you start your server with server.start(). You always have to start your server immediately after adding the service else it will throw an error.

Step 3: Create the client

With our server halfway done, let’s create a client that will be invoking our functions on the server.

Create a client.js file in the project directory with the following code:

Same packages we have to require too, and we also have to load our .proto file. We are connecting to the server with the same address; it could be 0.0.0.0 or localhost as you wish.

The first method we invoke is the createBook method which according to our definition schema in the .proto file takes in an object containing the id and text. A callback function is called with either an error or a response gotten from the server.

The same goes for the other two methods we invoke. The readBooks method sends no input if you remember clearly.

Step 4: Complete the server

Let’s take a look at our server.js again. At this point, we want to make the function receive and send data to the client as expected. With that in mind, let’s complete the business logic in the server.js file by replacing the noob functions in the server.js with the functions in the code below:

Take a look at the createBook method which has two parameters: call and callback. Note that all methods in gRPC always take in two parameters. The call contains an object which also has a request object containing values we will send from the client. And the `callback` is a way of returning values to the client as well.

We are using an array to mirror our database and the value from the request object is received and pushed to the book array. The second method readBooks works typically the same only that we have to find the element using the id value from the request object.

For a quick test, I had to run my debugger and also run my client.js file on the terminal.

You should see the responses on the terminal window. Just in case you do not want to use a debugger, I suggest you define the scripts in your package.json to be able to run them on individual terminals. Your scripts should be like this:

Now, to run the code open your terminal and run npm run grpc-server to start your server. Then open another terminal and run npm run grpc-client and you would get the results on your terminal.

Pros

Very performant and fast in data exchange because of its use of Protocol buffers.
Provides automatic code generation.
One client library works for all languages. Thus it is language agnostic.
It supports streaming both server and client.

Cons

Hard to implement.
No browser support because gRPC uses HTTP/2 underhood which our browsers currently do not work with. gRPC-web is presently available just in case you want to create a contract with your client which is the browser.

Conclusion

We discussed the concept of gRPC; how it is mostly used for integrated systems. The gRPC methods of communication: Unary, Server streaming, Client streaming and Bidirectional streaming are on a need-to-know basis when working with gRPC. We also looked at a basic implementation of a gRPC server and client. For more guides, I suggest you look at their Guides page.

Remote procedure calls are the backbone of distributed computing and gRPC has been able to cover the limitations without us reinventing the wheel.

While using gRPC, it is important to know that the best practices are still evolving, unlike a paradigm like REST where most of these have been standardized.

So although we all like to apply new technologies & upgrade our code using gRPC should be complemented with proper testing. And even if we feel we’ve tested everything before release it’s always necessary to verify that our users have a great experience with our product.

A solution like SessionStack allows us to replay customer journeys as videos, showing us how our customers actually experience our product. We can quickly determine whether our product is performing according to their expectations or not. In case we see that something is wrong, we can explore all of the technical details from the user’s browser such as the network, debug information, and everything about their environment so that we can easily understand the problem and resolve it. We can co-browse with users, segment them based on their behavior, analyze user journeys, and unlock new growth opportunities for our applications.

There is a free trial if you’d like to give SessionStack a try.

If you missed the previous chapters of the series, you can find them here: