Supercharge your REST APIs with Protobuf

JSON (JavaScript Object Notation) has been the go-to data interchange format when it comes to REST APIs. Long ago developers ditched XML in favor of JSON because JSON was compact, Schema-less, human readable and easy to transfer on wire.

The schema-less nature of JSON ensures that you can add or remove fields and you’ll still have valid JSON. But this also means that now, your once fully functioning clients will start failing because of added or removed fields. This problem magnifies when you have a micro-services architecture and have 100s (if not 1000s) of services talking to each other over JSON and you accidentally change the JSON response of one of the service.

Also, JSON takes unnecessary extra space by repeating field names (in case you are using Arrays) and becomes quite non-human-readable once you start nesting your data structure.

In 2001, Google developed an in-house, platform and language independent ,data serialization format named Protobuf (short for Protocol Buffers) to deal with all the shortcoming of JSON. The design goals of Protobuf were simplicity and speed.

In this post I’m going to share what are Protobufs and how replacing JSON in your REST APIs can significantly simplify the data serialization between client and server.

Table of Content

1. What’s and How’s of Protobuf
2. Tooling
3. Protobuf definition
4. Creating a REST endpoint
5. Consuming the REST endpoint
6. Bonus — Comparing with JSON
7. Conclusion

1. What’s and How’s of Protobuf

The Wikipedia for Protobuf says :

Protocol Buffers (Protobuf) is a method of serializing structured data. It is useful in developing programs to communicate with each other over a wire or for storing data. The method involves an interface description language that describes the structure of some data and a program that generates source code from that description for generating or parsing a stream of bytes that represents the structured data.

In Protobuf the developer defines a data structure (called messages) in a .proto file which then compiles to code with the help of protoc compiler. This compiler comes with code generators for multiple languages (both from google and community) and generates data structure to store the data and method to serialize and de-serialize them.

Protobuf messages are serialized into a binary format rather than text like JSON, hence the messages in Protobuf are not at all human-readable. Due to binary nature Protobuf messages can be compressed and takes less space than an equivalent JSON message.

Once you are done implementing the server, you can share the .proto file (like you share the schema of JSON that your API expects and returns) with the clients and they can leverage the same code-generation to consume the messages.

2. Tooling

We need to install following tools to follow the tutorial.

1. ) VS Code or your favorite code editor.
2. Golang compiler and tools (we’ll be writing our server and client in Go)
3. protoc the protobuf compiler.

Follow the installation instructions for each individual tool. I’m skipping the instructions here for the sake of brevity but do let me know if you face any errors, I’ll be happy to help.

3. Protobuf Definition

In this section, we’ll create a .proto file which we’ll use through out this demo. This proto file will have two messages EchoRequest and EchoResponse.

We’ll then create the REST endpoint accepting EchoRequest and reply back with EchoResponse. Then we’ll create a client (also in Go), consuming the REST endpoint.

Before I start , I want you to note few things about directory structure of this project.

1. I have created a folder github.com/kaysush in $GOPATH/src folder. $GOPATH variable gets set when you install go compiler and tools.
2. I’ve put my project folder protobuf-demo in github.com/kaysush.

You can see the directory structure in the illustration below.

$GOPATH
    ├── bin
    ├── pkg
    └── src
        └── github.com
            └── kaysush
                └── protobuf-demo
                    ├── server
                    │   └── test.go
                    ├── client
                    └── proto
                        └── echo
                            ├── echo.proto 
                            └── echo.pb.go

Create a echo.proto file.

Compile the proto file to golang code.

protoc echo.proto --go_out=.

This will generate a echo.pb.go file, which has go code with our messages defined as struct.

As a test we’ll see if marshaling and un-marshaling of our messages is working correctly.

Execute it.

go run test.go

You should see following output.

Value from un-marshalled data is Sushil

This shows that our Protobuf definition is working fine. In the next section we’ll implement a REST endpoint and accept a Protobuf message as payload of the request.

4. Creating a REST endpoint

Golang’s net.http package is self sufficient for creating REST APIs but to make our a bit easier we’ll be using gorilla/mux package for implementing our REST endpoint.

Install the package with following command.

go get github.com/gorilla/mux

Create a server.go file in the server folder and lets start coding.

The current directory looks like this.

$GOPATH
    ├── bin
    ├── pkg
    └── src
        └── github.com
            └── kaysush
                └── protobuf-demo
                    ├── server
                    │   ├── test.go
                    │   └── server.go
                    ├── client
                    └── proto
                        └── echo
                            ├── echo.proto 
                            └── echo.pb.go

The code for the Echo function should be straightforward to understand. We get the http.Request from which we read the bytes using iotuil.ReadAll, post which we Unmarshal those bytes into EchoRequest and read the Name field.

Then we follow the same steps in reverse to construct a EchoResponse.

In the Main() function, we define a route /echo which should accept POST method and handle the request by calling Echo function.

Start the server.

go run server.go

You should see the message Starting API server...

Your REST-ish API (because we are not following REST specification for POST requests) with /echo endpoint accepting POST is ready to accept Protobuf messages from clients.

5. Consuming the REST endpoint

In this section we’ll implement our client which consumes the /echo endpoint.

We have both client and server in the same code base hence we do not need to re-generate the code from proto files. In real world usage, you’ll share your proto file with the client which will then generate its code files in the programming language of its choice.

Create a client.go file in client folder.

The client should be even more straightforward to understand. We are using http.Post to send the Protobuf bytes to our API server and read back the response and then Unmarshal it to EchoResponse.

Run the client now.

go run client.go

You should see response from server.

Response from API is : Hello Sushil

6. Bonus — Comparing with JSON

We have successfully implemented the API which consumes Protobuf instead of JSON.

In this section we’ll implement an endpoint which accept the similar EchoJsonRequest in JSON and responds back in JSON as well.

Let’s implement another package with my structs for JSON.

Then I’ll add new function to our server.go.

Add binding for this new function in main().

r.HandleFunc("/echo_json", EchoJson).Methods("POST")

Lets modify the client to send repeated requests to both Protobuf and JSON endpoints and calculate the average response times.

Run both server and client.

Our server logs the content length of the request and as you can see the Protobuf request is 8 bytes while the same JSON request is 17 bytes.

JSON has double the request size for a trivial message

The client logs out the average response time in nano-seconds(Marshalling the Request + Sending Request + Unmarshalling the response) for both the Protobuf and JSON requests.

I ran the client.go 3 times and although the difference in average response time is quite small, we can see it the Protobuf request has consistent smaller average response time.

The difference is small because our message is quite small, as you increase the message size the cost of un-marshaling it into JSON increases.

Multiple comparative requests

7. Conclusion

Using Protobuf instead of JSON in your REST APIs can lead to smaller request sizes and faster response times. In our demo the response time effect was not starkly visible because of small payload size, but seeing the pattern, it is safe to say that Protobuf should perform better as opposed to JSON.

And there you have it. Replaing JSON with Protobuf in your REST APIs.

In case you find any issue with my code or have any question, feel free to drop a comment.

Till then happy coding! :)