Protocol Buffers & gRPC
What is Protocol Buffers?
It is an implementation that is used to platform and language independently serializing and parsing structured data in many areas such as communication protocols and data storage. Since its use in communication protocols is similar to the use of XML, we can proceed by comparing it with XML. Faster, simpler and smaller than XML.
Messages are defined once and can be used in various flows. (Message Definition, Interface Definition Language). This is pretty similar with data structure definition in C programming language. Message definition contains key-value pairs that must have a unique id.
We can examine the following example.
The message format is simple and understandable as can be seen in the example. Each field contains a value and a type such as string, integer, boolean, float etc. Bytes (string, int32) and other protocol message types allows structuring data hierarchically. Optional fields, required fields and repeatable fields can be defined. That way simple accessors (like name() and set_name() ) are able to serialize/parse structured data.
Why use protocol buffers instead of XML?
Protocol buffer messages have many advantages over XML for serializing structured data.
- Simple
- 3 to 10 times smaller
- 20 to 100 times faster
- Uncertain states are less possible than XML
- Easier to define programmatically
- The above Developer protocol buffer message will took 100–200 nanoseconds to convert to binary and will be 28 bytes long
- XML version of the same data will take 5000–10000 nanoseconds and will be 69 bytes.
Summary:
- New fields can be defined easily and middlewares can server raw data directly.
- There might be situations that serialization and parsing are avoided. Simple accessors doing that job automatically will eliminate that need.
- Beside the use of request with short life span, useful data storing has made protocol buffer widely used.
- Used for communications protocols and storing data.
Let’s compile the above file with protoc (proto compiler). We need to compiler plugin for the language we selected. (Python)
First we need to install this compiler on our system. (See doc: https://github.com/protocolbuffers/protobuf/tree/master/python)
It is enough to run the following command to compile. (Assuming that we are in the same directory as the proto file)
$ protoc --proto_path=. --python_out=.
This command will be created file proto_buffer_sample_pb2.py in same directory. An you can see developer class if you look at the created python file.
The first version of proto buffers was written by google developers for use in internal systems. But it was not turned into an open source. Because there was too much redundant code in it, and the generic solutions lacked version two. At the time of the version, all these problems were solved and presented to the world as an open source. The third version is now in beta.
RPC (Remote Procedure Call)
After briefly summarizing proto buffers, we can talk about the remote procedure call. The remote procedure call is not actually a structure connected to proto buffers. RPC is a protocol that one program can use to request a service from a program located in another computer on a network without having to understand the network’s details. A procedure call is also sometimes known as a function call or a subroutine call. The client-server process is based on a structure. The client calls a method and the service provides this method.
What is gRPC?
We can talk about grpc by combining the above two issues. gRPC is a service architecture based on protocol buffers that was built by Google and it is also an RPC framework.
C, Java and Go implementations are available. Grpc libraries developed for languages such as Python, Javascript, and Ruby are based on C implementation. It only works on HTTP 2.0 and supports Server Push. gRPC was released by google in 2015. It is known that a large part of google internal services have been implemented with grpc. Some companies such as docker, netflix, cisco also use grpc in their internal services.
gRPC is an RPC framework, the working principles are the same as RPC. We need to explain in detail the following way.
1. The client method calls from client stub as a local method. The client stub resides within the client’s own address space.
2. The client stub marshalls(pack) the parameters into a message. The client stub makes a packaging process to standardize this method call and transmit it over the network. This packing process is called marshalling.
3.The packet that is ready to be sent to the server is forwarded to the transport layer by the client stub.
4. On the server, the transport layer passes the message to a server stub, which demarshalls(unpack) the parameters and calls the desired server routine using the regular procedure call mechanism.
5. When the server procedure completes, it returns to the server stub, which marshalls the return values into a message. The server stub then hands the message to the transport layer.
6. The transport layer sends the result message back to the client transport layer, which hands the message back to the client stub.
7. The client stub demarshalls the return parameters and execution returns to the caller.
We will make an example for gRPC to be understood, but a brief information about HTTP 2.0 will be useful before this.HTTP 2.0
HTTP 2.0
HTTP 2.0 was released in 2015. It is faster than HTTP 1.1 and prevents client side delays. HTTP 2.0 survives through a persistent connection. This means that the handshake phase is not repeated during each request.
Multiplexing, Compression and Server Push supports and so runs faster.
The transmission of data is based on text in HTTP 1.1, but is provided in binary in HTTP 2.0. This eliminates the cost of serializing data in the intermediate layers. At the same time, encryption has become the standard on the browser side, which allows for a more secure data stream in HTTP 2.0. For more detail and comparing with HTTP1.1 see doc: https://medium.com/@factoryhr/http-2-the-difference-between-http-1-1-benefits-and-how-to-use-it-38094fa0e95b
gRPC Example With Nodejs
We will need both server and client to be able to do gRPC sample application. We will write both client and server in this project. The scenario will be like this. A company is looking for new developers for backend and frontend teams. There are certain criteria depending on the position and these criteria have a certain score. If the total score is above 10, the candidate will be eligible for the work.
First, we need to define the message type, the service and the methods of this service.
We need to write the message types, the service definition and the method definitions within the service in the proto file of our project. Because proto compiler libraries are written on the compilation of proto extension files.
service EligibleCandidateForWorkService {} In this line we will define the service.
The information we will write in this scope will show us which methods of this service, what parameters these methods take and what the return values are. Looking at the example above, we see the line rpc EligibleForWork (Candidate) returns (WorkEligibility) this line tells us that the method is EligibleForWork, this method has received a parameter type Candidate and the return value is returned to the value of WorkEligibility.
We made our message definitions as seen in the following sections. we made these definitions as proto-buf message.
Next step; to implement our service here. Below we can see the server’s index.js file.
Server application is ready to connect from 50050 port.
Let’s write the client application that will connect to this server application.
const client = new proto.eligible_work.EligibleCandidateForWorkService('localhost:50050', grpc.credentials.createInsecure()); line is for connecting to grpc server. This connection is a persistent connection so we don’t need to reconnect each request.
I’ll write a post that will compare grpc with rest. We will perform performance tests between standard rest and grpc application.
All helper methods and project details:
References and resoruces:
