Understanding Protocol Buffers — protobuf
Intro
Protocol Buffers (protobuf) is a method of serializing structured data which is particulary useful to communication between services or storing data.
It was designed by Google early 2001 (but only publicly released in 2008) to be smaller and faster than XML. Protobuf messages are serialized into a binary wire format which is very compact and boosts performance.
Details
This protocol involves a specific interface description language to describe the data structure defined in a .proto file.
A program from any language supported is able to generate source code from that description in order to create or parse a stream of bytes that represents the structured data.
Protocol buffers can serve as basis for a remote procedure call (RPC) which are widely used for inter-machine communication, most commonly used with gRPC. Protobuf is similar to Apache Thrift used by Facebook, Ion by Amazon or Microsoft Bonds Protocol.
Example
In this example we will be creating two projects:
- A Java console application that will use the customer .proto specification to generate a file with a hard coded customer
- A C# console application that will read the hard coded customer file generated by Java console application, and display the data in console
tl;dr
If you just want to read some code and figure it out on your own, I've setup these two applications repository. First follow instructions of the README file of the Java project repository:
Then after generating the protobuf serialized file, follow the instructions of the README file of the C# project repository:
Protobuf Contract
First of all, let's create a structure that will represent a customer. The data required for a customer are:
- Unique identification
- Photo
- Name
- Birthdate
- Creation timestamp
- Last update timestamp
So, we need to create a .proto file like this:
Some notes about the code above:
- Timestamp type is a "Well Known Type" introduced in proto3, you can use these types by importing in the first lines of proto file
- Date and Money types are a "Google Common Type", differently than "Well Known Type" you are not able to use it only by importing. You have to copy these types definitions file from google repository and paste it in your project whatever the language you are using.
There are other scalar types, you can read the documentation here. Well Known Types here or Google Common Types here.
This .proto file will be a common resource in C# and Java but for simplicity sake, I'll recreate it in both projects repository. The ideal for big and complex projects is to have a separate repository as a neutral ground for projects.
Java Console Application
That said, let us start by creating the Java console application. For this example I'll be using OpenJDK 15, IntelliJ IDEA CE and Maven as build tool to do so.
- Open IntelliJ IDEA CE and choose to create a new project
2. Choose Maven in the left panel, select your Java JDK at top of right side. As said I'll be using OpenJDK 15 previously installed
3. Fill the next fields as you wish, for example:
Project created, let's start by adding project source encoding, telling maven compiler we are using JDK 15 and adding protobuf-java dependency by Google. Add the following lines to pom.xml under "project" tag, right after "version" tag:
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>15</maven.compiler.source>
<maven.compiler.target>15</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>3.13.0</version>
</dependency>
</dependencies>
Now let's include the .proto files defined in the section above. But before create a proto directory in src/main:
Then add a new file named: customer.proto to it and add the code mentioned in above section:
The money.proto and date.proto imports will be pointing error, because we did not created it yet. You can create then repeating the process above and adding the of money.proto and date.proto from Google repository.
Ok, protobuf contracts created, now, to generate code (java classes) from this contract, we need to use protoc executable to compile .proto files targeting the desired output language. There are two main ways of doing this:
- Manually, by downloading protoc in your machine and running it. If you wish to proceed with this method read protoc installation guide here
- Automatically, by adding protoc code generation to your maven project build. There are several maven plugins for this, but I will be using protoc-jar-maven-plugin, that wraps protoc executable as a jar so it can run in any OS and compile your .proto files.
You can use it by adding these lines to your pom.xml under "project" tag, right after "dependencies" tag:
<build>
<plugins>
<plugin>
<groupId>com.github.os72</groupId>
<artifactId>protoc-jar-maven-plugin</artifactId>
<version>3.11.4</version>
<executions>
<execution>
<id>protoc.main</id>
<phase>generate-sources</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<protocVersion>3.13.0</protocVersion>
<addSources>main</addSources>
<includeMavenTypes>direct</includeMavenTypes>
<includeStdTypes>true</includeStdTypes>
<includeDirectories>
<include>src/main/proto</include>
</includeDirectories>
<inputDirectories>
<include>src/main/proto</include>
</inputDirectories>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
Always check for newer stable versions before adding dependencies or plugins to your pom.xml
We are also including the proto directory structure so the plugin can know where the .proto files are, and compile them.
At this point you might want to run protoc compiler to generate the java protobuf contract classes so, click maven tab and "run maven goal" button and run a mvn clean install
If is successfully compiled, this message will pop up:
Next, create a package "dev.danielpadua.protobufexamplejava" and put inside a main class for our console application:
If you type "Customer" IntelliJ autocompletion will appear and suggest to import the class that was generated by protoc via plugin:
So, it all went well. Now we can write the code to generate a hard coded customer and write it to a file in a directory you want (inside main method):
Date birthdate = Utils.toGoogleDate(LocalDate.of(1990, 4, 30));
Money balance = Utils.toGoogleMoney(BigDecimal.valueOf(9000.53));
Timestamp createdUpdateAt = Utils.toGoogleTimestampUTC(LocalDateTime.now());
String fullPath = "/Users/danielpadua/protobuf/protobuf-customer";
try (FileOutputStream fos = new FileOutputStream(fullPath)) {
Customer daniel = Customer.newBuilder()
.setId(1)
.setPhoto(ByteString.EMPTY)
.setName("Daniel")
.setBirthdate(birthdate)
.setBalance(balance)
.setCreatedAt(createdUpdateAt)
.setUpdatedAt(createdUpdateAt)
.build();
daniel.writeTo(fos);
System.out.println("protobuf-customer created successfully");
} catch (FileNotFoundException e) {
System.out.println(format("could not find file {0}", fullPath));
} catch (IOException e) {
System.out.println(format("error while reading file {0}. exception: {1}", fullPath, e.getMessage()));
}
Note the code above uses a simple Utils class to convert between Java LocalDate and Google Date, Java BigDecimal and Google Money and Java LocalDateTime and Google Timestamp. You can add my Utils class to your project using the following code:
If you have the following problem in IntelliJ, you can simply use the suggested solution "Set language level to 8 — Lambdas, type annotations etc"
Now, let's run the application. Click "Add Configuration" button located at top right, right after build button. Click the plus icon and select "Application":
Then fill the configuration name and select the main class to run:
After, just click run button or debug it if you want:
A successfully result should show the following message:
Now check the directory you defined for output file:
We successfully implemented a simple Java console application that creates a protobuf message using the structure defined in a .proto file, using automated protobuf compilation with maven build tool. Now to prove it is useful to different languages communication, we'll create a C# console application to read this file and show in console.
C# Console Application
For the C# example I'll be using: .NET 5, Visual Studio Code and Grpc.AspNetCore package that has a built-in protobuf compiler that can be attached to dotnet build.
Open Visual Studio Code, open a new terminal, navigate to a directory you want to keep your project and execute the following commands one at a time:
mkdir protobuf-example-csharp
cd protobuf-example-csharp
dotnet new console -o src/DanielPadua.ProtobufExampleCsharp
dotnet new xunit -o tests/DanielPadua.ProtobufExampleCsharp.Tests
dotnet new sln -n DanielPadua.ProtobufExampleCsharp
dotnet sln add src/DanielPadua.ProtobufExampleCsharp/DanielPadua.ProtobufExampleCsharp.csproj
dotnet sln add tests/DanielPadua.ProtobufExampleCsharp.Tests/DanielPadua.ProtobufExampleCsharp.Tests.csproj
All set, project created, now let's open it in Visual Studio Code:
The project structure must look like this:
Click "Yes" in the message at bottom right, for Omnisharp create .vscode folder with assets to run/debug project. Select "DanielPadua.ProtobufExampleCsharp":
Open terminal again at the src/DanielPadua.ProtobufExampleCsharp level, and run:
dotnet add package Grpc.AspNetCore
Now let's include .proto contracts. Create a directory under main project root named: "Protos" and create the .proto listed in above section:
Add the .proto files in .csproj for protoc compile when dotnet build runs:
Now let's build the project to generate C# compiled classes from .proto contracts. In terminal run:
dotnet build
A successful output message must look like this:
Next, replace the "Hello World" in Main method for the following lines:
var fullpath = @"/Users/danielpadua/protobuf/protobuf-customer";
using var inputStream = File.OpenRead(fullpath);
Customer c = Customer.Parser.ParseFrom(inputStream);
Console.WriteLine("Customer from protobuf-example-java:");
Console.WriteLine(c.ToString());
Now you will have to import Contracts namespace, press ctrl+.
(windows, linux) or cmd+.
(macOs) to open autocomplete, and the import option will appear:
If it does not appear, try to restart Omnisharp using: ctrl+shift+p
(windows, linux) or cmd+shift+p
(macOs), type: restart omnisharp
and hit enter:
Sadly the protobuf compiler and Omnisharp integration is not perfect, but it works.
Make sure you are reading the same directory and file that java generate the file, and then run the C# console application by hitting run button (if you configured run/debug assets correctly) or simply by running a: dotnet run
being at main project root directory:
And we did it, received and interpreted a protobuf serialized message generated by a Java application in a C# application.
Conclusion
Protobuf was made to be faster, lighter hence better performing than other protocols. So, do a quick search like: "protobuf vs json performance" or other, there are tons of benchmarks and success cases.
In this article I hope to have given a dive in for those who, like me some time ago never even heard of protobuf and always sticked to JSON and XML.
See you soon!
Originally published at https://blog.danielpadua.dev on November 24, 2020.