A cute bazel + proto hack for golang IDEs

If you haven’t noticed I have developed quite a liking to building golang with bazel. So much so that the first two posts of this publication have been about how to do golang monorepo.

When I was exploring writing golang + protobufs and compiling with bazel, much like most people I came to realize that my editor will not work with bazel generated code. This post explores my workaround for that, it is a hack so use at your own peril.

UPDATE: I wrote a library for this solution here

A tiny gopher hack with questionable existence

Problem Statement

Here is a GitHub issue that talks about the problem, and I will try to summarize it. In terms of code we are starting off at part 2 of golang monorepo series

Golang monorepo part 2

Last time, we wrote hello.proto and included the import github.com/nikunjy/protos/hello in our go code. This worked because behind the scenes bazel generates hello.pb.go using proto compiler. This generated file is placed in bazel-bin folder and you now have a problem that your IDE doesn’t work as it would for regular go code because your project is no longer go build able

And right now there is no good solution for this problem so I present to you this hack I used.

Solution Summary

For those who prefer reading code more than reading english you can clap at this story and go to this commit. 😸

My solution leverages two things:

1. Generated files from proto files are accessible in go_generated_srcs on the target go_proto_library . So I can write a bazel target called proto_link which would copy the generated source into the folder where proto file was.
2. I can write a small Gazelle extension to add proto_link target to all BUILD files where I have go_proto_library . This will automate away the job of me placing it manually and so I will just have to run bazel run //:gazelle to make sure whenever I add more proto code my BUILD files will have the right contents.

Ok…I need more details

Gazelle Extension

As we talked about in golang monorepo part 1, gazelle is this super powerful magical tool that can generate bazel BUILD files for you.

How gazelle works is outside the scope of this post but basically it has concept of a language, and go and proto are two languages implemented by default. How do they work ? the implementation can literally read .go and .proto files and figure out dependencies and generate BUILD files for your code.

Gazelle is also extensible, you just need to implement this interface, inform bazel build system and bazel will run it when you run bazel run //:gazelle like we were doing in the previous posts. Gazelle library provides you with bunch of helpful metadata using which you can write your implementation, here are more details and examples.

Reading go_generated_srcs

Here is the logic for copying over the generated sources.

Line 4, get the generated srcs for the go_proto_library for hello.proto

Line 13, basically string interpolation of copy_into_workspace.sh it ends up being basically the cp command from line 7

And that is it, the whole implementation is here

Stitching Together

Now that we have written a gazelle extension which is going to place a target go_proto_link in each BAZEL file where there is a proto. And we have an implementation of go_proto_link we can tell bazel build system to run it

This should be fairly obvious, we declared a gazelle_binary on Line 4 which knows about the default languages and the one that we have written.

Line 11 is the target that gets invoked when you run bazel run //:gazelle

Demo

Run bazel run //:gazelle

Awesome, this generated the go_proto_link target in protos/hello/BUILD.bazel . Now you can write a script that queries for all such targets and runs them, that is easy

bazel query 'kind("proto_link", //...)'  | xargs bazel run

And you are done. Whenever you now update a proto file or add a new one, you can run this script and be sure that you will have the updated auto generated code.

Use it in your project

UPDATE: You can directly use this library I wrote. Follow installation instructions there

nikunjy/golink

Note

Your IDE and invoking go build will only work if your proto files follow the same folder structure as you expect the import to be. So for example if my hello.proto had optional go_package = "github.com/nikunjy/go/blahblahhello" at the top of the file then this hack would not work because the script would copy it in the source folder which is go/protos/hello thus making it accessible by import github.com/nikunjy/go/protos/hello but the bazel build system which is how you are building and publishing images would expect the import to be github.com/nikunjy/go/blahblahhello