Protobuf Automation

When we started a development project 3 years ago, the technology stack was more or less defined by previous projects within the company and team skills, as is usually the case in startups. One of the technologies from that stack was Protocol buffer, which implements GRPC. Surprisingly, it turned out to be a really useful technology which saved us a lot of time and effort, but that is another story!

What is Protobuf

Its owner, Google, describes it as alanguage-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. So, in fact, you have some ‘proto language’, which is pretty strict, and you need to describe messages and/or endpoints of your services with this language.

syntax = "proto3";

package foo;

service BarService {
  rpc SomeMethod(MyMethodRequest) returns (MyMethodResponse) {}
  rpc AnotherMethod(AnotherMethodRequest) returns (AnotherMethodResponse) {}
}

message MyMethodRequest {
  string some_parameter = 1;
  int64 some_other_parameter = 2;
}

message MyMethodResponse {
  repeated float array_of_floats = 1;
}

message MessageThatICanReuse {
  float min = 1;
  float max = 2;
}

message AnotherMethodRequest {
  string some_id = 1;
}

message AnotherMethodResponse {
  MessageThatICanReuse price_range = 1;
  MessageThatICanReuse area_range = 2;
}

After that, you will be able to compile this file to various languages across your application. You will get a number of generated files ready to use as a client or a server.

How we automated proto compilation to get rid of manual work

Most of the time, I saw how people were compiling proto files locally using git submodules to retrieve them and, to be honest, we were doing it in the same way. But after some time we came up with the idea to automate it more than just using Makefiles. We created a separate repository for generated files to use it as a go, python package, PHP library or even node module. Of course, you can’t just use generated files in repo for all of this, so you need to add a few files to make it work. I’m going to add more details about how it works and how to implement it elsewhere.

Build docker images for every language you need

As I said above, we had go, python, php and js, so I’m going to provide images that we are using to compile proto files for our stack. We tried to make the file sizes as tiny as possible to speed up the compilation stage.

# Golang

FROM golang:1.17.0-alpine as builder

ENV GO111MODULE=on
RUN go get -u google.golang.org/protobuf/cmd/protoc-gen-go google.golang.org/grpc/cmd/protoc-gen-go-grpc

FROM alpine

RUN apk --no-cache add ca-certificates protoc protobuf-dev

COPY --from=builder /go/bin/protoc-gen-go /usr/local/bin
COPY --from=builder /go/bin/protoc-gen-go-grpc /usr/local/bin

# PHP

FROM alpine as builder

RUN apk --no-cache add ca-certificates protoc protobuf-dev git cmake make gcc g++

# depends on grpc version that you need ofc
RUN git clone -b v1.30.0 https://github.com/grpc/grpc /grpc
RUN cd /grpc && \
    git submodule update --init && \
    mkdir -p cmake/build && \
    cd cmake/build && \
    cmake ../.. && \
    make protoc grpc_php_plugin

FROM alpine

RUN apk --no-cache add ca-certificates protoc

COPY --from=builder /grpc/cmake/build/grpc_php_plugin /grpc_php_plugin
COPY --from=builder /usr/include /usr/include

# Python

FROM python as builder

WORKDIR /src

RUN apt-get update && apt-get install -y ca-certificates git-core ssh

# unfortunatelly we didn't find more lightweight option
RUN pip install grpcio-tools==1.36.1

# Web (JS)

FROM alpine

WORKDIR /src

RUN apk add --update ca-certificates curl protoc protobuf-dev

# version should be compatible with your library version
RUN curl -sSL https://github.com/grpc/grpc-web/releases/download/1.2.1/protoc-gen-grpc-web-1.2.1-linux-x86_64 \
    -o /src/protoc-gen-grpc-web && chmod +x /src/protoc-gen-grpc-web && \
    mv /src/protoc-gen-grpc-web /usr/local/bin/protoc-gen-grpc-web

And how to compile proto using them

# Golang (go-build.sh)

for package in foo, bar; do \
  protoc \
      --proto_path="$1" \
      --go_opt=paths=source_relative \
      --go_out="$2" \
      --go-grpc_out="$2" \
      --go-grpc_opt=paths=source_relative \
      "$1"/${package}/*.proto && \
      ls "$2"/${package}/*.pb.go; \
done

# PHP (php-build.sh)

for package in foo, bar; do \
  protoc \
      --proto_path="$1" \
      --php_out="$2" \
      --grpc_out="$2" \
      --plugin=protoc-gen-grpc="/grpc_php_plugin" \
      "$1"/${package}/*.proto;
done

# Python (python-build.sh)

for package in foo, bar; do \
  python -m grpc_tools.protoc \
      --proto_path="$1" \
      --python_out="$2" \
      --grpc_python_out="$2" \
      "$1"/${package}/*.proto && \
      ls "$2"/${package}/*.py | xargs -n1 -IX bash -c "sed s/'from ${package}'/'from ..${package}'/ X > X.tmp && mv X{.tmp,}"; \
  touch "$2"/${package}/__init__.py
done
# for using repo as a python package
touch "$2"/__init__.py

# Web (JS) (web-build.sh)

for package in foo, bar; do \
  protoc \
      --proto_path="$1" \
      --js_out=import_style=commonjs,binary:"$2" \
      --grpc-web_out=import_style=commonjs,mode=grpcwebtext:"$2" \
      "$1"/${package}/*.proto && \
      ls "$2"/${package}/*.js; \
done

And, finally, as an example of how to combine all these elements into something working, I’m going to share our cloud build file, which is applicable for GCP only, but you can migrate things that you need to any Cl tool ofc.

steps:
  # clone proto repo
  - name: 'gcr.io/cloud-builders/git'
    id: clone-repo
    args:
      - clone
      - --single-branch
      - --branch
      - $BRANCH_NAME
      - git@github.com:your-proto-repo.git
      - /workspace/proto

 # keep commit msg to use it for target repo commit
 - name: 'gcr.io/cloud-builders/git'
    id: save-commit-msg
    entrypoint: 'bash'
    args:
      - '-c'
      - |
        cd /workspace/resolute-platform-proto && \
        git log -1 --pretty=format:'%an: %s' >> /workspace/commit-message.txt
    volumes:
      - name: 'ssh'
        path: /root/.ssh
    waitFor: [ 'clone-repo' ]

 # linters
  - name: 'yoheimuta/protolint'
    id: linter
    args: [ "lint", "." ]
    waitFor: [ 'clone-repo' ]

 # clone repo to store generated files
  - name: 'gcr.io/cloud-builders/git'
    id: clone-gen-repo
    args:
      - clone
      - --single-branch
      - --branch
      - dev
      - git@github.com:res-am/proto-gen.git
      - /workspace/proto-gen
    volumes:
      - name: 'ssh'
        path: /root/.ssh
    waitFor: [ 'linter' ]

 # compile into go files
  - name: 'YOUR GO PROTO IMAGE FROM EXAMPLES ABOVE'
    id: build-golang
    entrypoint: 'sh'
    args:
      - '-c'
      - |
        rm -rf /workspace/proto-gen/go && \
        mkdir /workspace/proto-gen/go && \
        /workspace/proto/go-build.sh /workspace/proto /workspace/proto-gen/go
    waitFor: [ 'clone-gen-repo' ]

 # compile into python files
  - name: 'YOUR PYTHON PROTO IMAGE FROM EXAMPLES ABOVE'
    id: build-python
    entrypoint: 'bash'
    args:
      - '-c'
      - |
        rm -rf /workspace/proto-gen/python && \
        mkdir /workspace/proto-gen/python && \
        /workspaceproto/python-build.bash /workspace/proto /workspace/proto-gen/python
    waitFor: [ 'clone-gen-repo' ]

  - name: 'YOUR WEB PROTO IMAGE FROM EXAMPLES ABOVE'
    id: build-web
    entrypoint: 'sh'
    args:
      - '-c'
      - |
        rm -rf /workspace/proto-gen/web && \
        mkdir /workspace/proto-gen/web && \
        /workspace/proto/web-build.sh /workspace/proto /workspace/proto-gen/web
    waitFor: [ 'clone-gen-repo' ]

  - name: 'YOUR PHP PROTO IMAGE FROM EXAMPLES ABOVE'
    id: build-php
    entrypoint: 'sh'
    args:
      - '-c'
      - |
        rm -rf /workspace/proto-gen/php && \
        mkdir /workspace/proto-gen/php && \
        /workspace/proto/php-build.sh /workspace/proto /workspace/proto-gen/php
    waitFor: [ 'clone-gen-repo' ]

 # add extra file for python package, commit and push changes
  - name: 'gcr.io/cloud-builders/git'
    entrypoint: 'bash'
    args:
      - '-c'
      - |
        cd /workspace/proto-gen && \
        git status && \ # more like debug information in build log
        git add -A && \
        git commit -m "$(cat /workspace/commit-message.txt)" && \
        git push && \
        bash autotag.bash && \ # increment git tag
        git push --tags
    waitFor: [ 'build-golang', 'build-python', 'build-web', 'build-php' ]

This pipeline will give you a simple way to use generated protobuf structures as a package, so you will be able to add it in requirements (python), go.mod (golang), composer (php) or package.json (js) just as an another library from your list.

P.S. Actually, if you want to use it as an npm package in js, you can create npm package additionally and push a new version for every new commit using your Cl. We used github npm registry and github actions for it.

So, what can this solution give you?

Pros

1. Versioning. It’s so useful when you can rollback to previous versions or understand how old proto file versions your service is using. In addition, you can always check which proto files were the source for this version. Protobuf is impressive in terms of compatibility. If you didn’t remove or change any fields, or didn’t change the numbering in the used message, your service won’t even feel a change. Adding or sometimes even removing fields without changing the numbering will not break your application in any way (except that you will stop receiving data from the deleted field, of course!)
2. This approach gives you more automation, so you save on wasting your valuable time too!
3. As it is part of the native programming language, It’s so native for your languages that you don’t need to set it up in anyway on your local machine or in CI.

Cons

1. For local development it would be considered bad practice to push changes into the main branch or your proto repo. So, without extra work, you could easily break some critical elements before your changes are even implemented.
2. You might have issues with hotfixes, because, for them, you need to make some changes at a specific point of time (for a stable release), which uses a specific tag for this kind of mechanism. You can’t push hotfix changes with the latest tag because, in this instance, you will also pick up new features.

Workarounds

1. You can create a separate CI pipeline for local development. It can be done via a generation of other branches as well. This way, if anyone pushes changes to a branch which is not a main branch, you can generate code with the same branch name in the destination repository. You will be able to use branch packages in your local dev environment and replace them with the latest tag when you prepare PR, although it’s not quite as convenient.
2. In this case, it is possible to create a tag with a hotfix suffix or a part of a new tag, and use it for services where you need to apply hotfix.

Final Words

As a result, we got a convenient mechanism that helps save valuable time. We continue to improve it with new variables in our day-to-day development, but the basic principle remains the same.