Protocol Buffer Shared Types

Inter-service communication with microservices is great, but also hard. If done incautiously it can quickly escalate into a wobbly Jenga tower where debugging mechanics and flow of messages bring more headaches than satisfaction.

Example of unstable architecture.

At Blokur our synchronous inter-service communication happens via gRPC, with the communication defined by Google’s protocol buffers.
To quote Google:

Protocol Buffers (a.k.a., protobuf) are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data.

Since they are language-neutral, protobufs can be used to generate code targeting a variety of languages. There are many libraries out there which can be used to achieve this. https://github.com/golang/protobuf and https://github.com/agreatfool/grpc_tools_node_protoc_ts are just two awesome examples.

Let’s see how to use protobufs and shared types to reduce redundancy!

How to use

Let’s assume that we want two services to exchange information about dogs (why not?) using protobufs. A simple setup can look like this:

These describe a gRPC server exposing the GetBorkersForOwner endpoint which returns a collection of dogs for the owner id specified in the request. Neat!

Since protobufs define how data is structured, it is imperative to have microservices using either the same definition or a backwards-compatible one.

This is why at Blokur we keep them in a single repository, called protorepo. It helps us make sure that communicating services are referring to the same definition (good versioning practices apply).

We also store the code generated with the protobufs in a _build folder so that the generated code is shared too (repos using protorepo do not need to generate their own version of the same code, which may differ depending on dependencies versions, build environment etc.).

protorepo can be structured like so:

protorepo/
├── Makefile
├── _build/
│   └── go/
│       └── kennel/
└── kennel/
    ├── index.proto
    └── kennel.proto

The handy Makefile has a target for generating the code:

protocol compiler is used to generate go code for each proto file in kennel

Reusing code

Now let’s say we also want to describe a notification service.

Quite similar to the previous example. However, in our fictitious dog-related tech product a user is also a dog-owner, therefore GetBorkersForOwnerRequestand GetNotificationsRequestsuddenly look very similar.

The issue in this example plus the impossibility in Protobuf of using primitive types as request or responses (they must be wrapped in specific message types we did in this example) introduces redundancy.

In this example the redundancy is not cumbersome but it could rapidly escalate with a more complex project. Since we don’t want to repeat ourselves, we can rectify our code by merging the two messages into a new type:

message User {
    int32 id = 1;
    // can be enriched
}

Which is stored insideptypes at the root of protorepo.
We also need to modify the Makefile allowing protocto reference ptypes during transpilation. This can be done with-I .:

protoc -I kennel -I . --go_out=plugins=grpc:_build/go/kennel “$(basename “$${filename}”)”

Alas, the generated go code will contain package reference issues due to misalignments in the produced files and paths between ptypes and other protobufs! Luckily, with a little inspiration from one of Google’s known types , a simple line addition can be enough:

go_package tells protoc what package path to use for the generated go file

Here is our final Makefilefunction for generating go code:

And the final folder structure is:

protorepo/
├── Makefile
├── _build
│   └── go
│   ├── kennel
│   ├── notifier
│   └── ptypes
├── kennel
│   ├── index.proto
│   └── kennel.proto
├── notifier
│   ├── index.proto
│   └── notifier.proto
└── ptypes
   └── index.proto

Now you have shared types in protobufs!

Feel free to explore this repo I made for you (with ❤️, obviously). It already contains the generated go files so that you can check out what the final output would be!