Create and Run a Protobuf Plugin

Learn how to create a protoc plugin.

Do you know what protocol buffers (protobufs) are? Do you know how to use them and do you know there are several extensions like gRPC and gRPC-web? Yet, you found a use case that is not supported out of the box, and even the list of third-party plugins is not covering your specific scenario?

A HelloWorld proto file

Sounds familiar? The best solution is to write your own protoc plugin. Protoc has powerful tools for this purpose, yet documentation on the internet for this is rare. I experienced this exact scenario. So instead of spending hours, trying to collect all the necessary information, let me tell you how it works.

Covering the full spectrum of possibilities when writing a plugin is too much for just one blog post. So this will be a series of several posts.

Picking a programming language

A protobuf plugin can be written in any language you like, as long as there is protobuf support for it. The easiest way is to use one of the languages that protoc supports out of the box (even though languages supported through plugins also work). Do you have a preference? Then go with that. Protoc itself is written in C++ so if you are familiar with that language, this could be a good choice for you.

In my personal experience, even though the output of the plugin is generated code, large parts of that code are static. In most cases, you still end up with dependencies on system or external libraries. There is no reason to generate code for things like lists or sets, so don’t re-invent the wheel and just import them instead. Even lines that are generated often follow a certain pattern. In the case of Java, you might have constant declarations that look like:

private static final String HELLO_WORLD_MESSAGE = "Hello World";

where only the name HELLO_WORLD_MESSAGE and value “Hello World” are actually generated.

So C++ is just not your thing? I recommend picking a language that has some level of support for templates or string-interpolation (such as Kotlin, Python or Typescript). For this tutorial, my language of choice is Python. It has a feature called f-strings that make the above example almost read like real code in the generator:

f'private static final String {name}_MESSAGE = "{value}";'

Other considerations are portability (Python code runs almost anywhere) and simplicity (many developers know Python or find it easy to learn).

Setting up the environment

Setting up protoc

First and foremost, you need the protobuf compiler (protoc) itself. The official instructions can be found on GitHub. However, they involve either compiling your own binary or manually download one.

I prefer using a package manager like brew or apt, since I think they are the easiest to use. In this case, pick the right command for you from below:

brew install protobuf # mac-users
apt install protobuf # linux-users

Setting up a virual environment

The code generator will depend on the protobuf library. To install it, create a new file called requirements.txt with the following content:

protobuf>=4.23.1

I recommend using a virtual environment instead of installing any dependencies globally. To do so and to install protobuf into that environment, run these commands:

python3 -m venv ./venv # Create the environment
source ./venv/bin/activate # Activate it
pip install -r requirements.txt # Install dependencies

A basic hello-world plugin

The basic hello-world plugin we’re about to create will do nothing more but creating a file called hello_world.txt with he content “Greetings, world!”. We will explore creating more complex output soon. For no, go ahead and create a new file called generator.py with the following content:

import sys

from google.protobuf.compiler.plugin_pb2 import CodeGeneratorResponse

if __name__ == "__main__":
    response = CodeGeneratorResponse()                    #1
    generated_file = response.file.add()                  #2
    generated_file.name = "hello_world.txt"               #3
    generated_file.content = "Greetings, world!"          #4
    sys.stdout.buffer.write(response.SerializeToString()) #5

In this script, we first create a new CodeGeneratorResponse (1), add a new file we want to generate (2), tell protoc how that file should be called (3), and finally specify the content of that file (4).

Lastly, we simply serialize the response to a string and write it to stdout (5). More on that later.

Running protoc with a plugin

To generate code using that plugin, create a simple proto-file called hello_world.proto:

syntax = "proto3";

/* A message to the world */
message HelloWorldRequest {
  string message = 1;
}

You can then generate code by running

mkdir generated                                      # 1
protoc -I./                        \                 # 2
       hello_world.proto           \                 # 3
       --python_out=generated      \                 # 4
       --hello-world_out=generated \                 # 5
       --plugin=protoc-gen-hello-world=generator.py  # 6

First, we are creating a new directory for the generated output (1). Then we call protoc by setting the proto include path to the current directory (set this to something else if you created hello_world.proto somewhere else) and specify the proto file we want to create (2 and 3).

In line (4), we’re setting the output directory for python code. This is optional, since we actually just want to call the plugin, but in most real-world cases you still want to generate the original protobuf.

In (5), we’re declare the output directory for a plugin called “hello-world”. And finally, we’re specifying where this plugin is located (6).

The value protoc-gen-hello-world=generator.py may seem a bit surprising. Protoc usually expects an executable called protoc-gen-<plugin-name> on the PATH. Since this is not the case here, we need to explicitly tell protoc, where the generator is located. In our case the script we created earlier.

If you now hit enter, you most likely receive a frienly error message:

generator.py: program not found or is not executable
Please specify a program using absolute path or make sure the program is available in your PATH system variable
--hello-world_out: protoc-gen-hello-world: Plugin failed with status code 1.

Protoc is not able to find the executable, because generator.py is not actually executable yet. To change that, run chmod +x generator.py, and add a hashbang at the beginning of generator.py:

#!/usr/bin/env python3

Now run the protoccommand again (you can skip the mkdir part). This time, no error-message should appear.

Look inside the generated directory you just created. You should find two files there, the hello_world_pb2.py that holds the python-version of the protobuf and your brand-new hello_world.txt.

Congratulations, you’ve written your first protoc plugin. 🎉

Two more things…

Remember that output to stdout? This is needed because protoc communicates with plugins via stdout/stdin. It’s one of the main reasons, why you can write a plugin in any language you like. All that’s required is a language that gives you access to those streams.

In this basic tutorial, we only created one file with a fixed output. However, the purpose of protoc is to generate code based on proto-files. To do so, stay tuned for the next post.

Finally, a running example of the code presented here is available on GitHub.

If you found this useful or interesting please like, share and subscribe.

David

Further Reading:

Distributed Tracing: A Guide for 2023

Microservices Monitoring: Cutting Engineering Costs and Saving Time

OpenTelemetry Tracing: Everything you need to know