Implementing gRPC communication

In the first version of BAG:EL, the system architecture consisted of:

• a coordinator node (written in Go)
• a number of worker nodes (written in Go)
• a client node (written in Go)
• RPC communication between these components using Go’s built-in RPC package. If you’re unfamiliar with RPC, I recommend watching this video. It also explains the differences between RPC and gRPC. In short, gRPC is an implementation of RPC developed by Google, and it uses a language-agnostic structure called protocol buffers to encode and send data over the network.

All of these components were implemented as Go command-line programs. One of the updates we wanted to make to our project BAG:EL was to build an actual client interface. I am the most familiar with building user interfaces using React, so I opted for React & TypeScript to build the new client. This update led to a new path for RPC communication. Since we were previously using Go’s built-in RPC between the client and the coordinator, but the client was no longer written in Go, I decided to move to a language-agnostic gRPC communication between the client and the coordinator.

Here are the steps I took to implement gRPC communication between our React frontend (“client”) and Go backend (“coordinator”):

• I started by defining a common .proto file to be used by *both* the client and the coordinator. View the file here. As for now, the client and the coord only communicate using a single method called StartQuery .
• I used the protoc compiler to generate the gRPC code on the client-side and server-side. That is, using the coord.proto file, the compiler generates .go files for coordinator to use, and .js files for client to use. The generated files are here and here respectively.
• This article explains that the browser (client) and the gRPC server (coord) need a proxy server to communicate: this is because the browser uses HTTP/1 whereas gRPC uses HTTP/2. Therefore, I setup an Envoy proxy server instance using Docker, which can be found here.
• Lastly, I needed to modify the client and coordinator code to use the new gRPC functions. For the client, this meant using the generated gRPC functions to make the startquery call to coord, see here. For the coord, it additionally meant creating a gRPC server instance (see here) and modifying the coord interface to match the interface required by the gRPC version of the startqueryfunction (see here and here).

After these changes, the new client and coord were able to communicate over gRPC. The beauty of it was that we moved to a brand new UI instead of our previous command-line client, and we didn’t need to make any changes to the existing RPC communication between coord and workers as coord was able to play the role of both a gRPC server and an RPC server.

The diagram below shows the communication between the components after moving to gRPC:

network diagram