A cute bazel + proto hack for golang IDEs
If you haven’t noticed I have developed quite a liking to building golang with bazel. So much so that the first two posts of this publication have been about how to do golang monorepo.
When I was exploring writing golang + protobufs and compiling with bazel, much like most people I came to realize that my editor will not work with bazel generated code. This post explores my workaround for that, it is a hack so use at your own peril.
UPDATE: I wrote a library for this solution here
Problem Statement
Here is a GitHub issue that talks about the problem, and I will try to summarize it. In terms of code we are starting off at part 2 of golang monorepo series
Last time, we wrote hello.proto
and included the import github.com/nikunjy/protos/hello
in our go code. This worked because behind the scenes bazel generates hello.pb.go
using proto compiler. This generated file is placed in bazel-bin
folder and you now have a problem that your IDE doesn’t work as it would for regular go code because your project is no longer go build
able
And right now there is no good solution for this problem so I present to you this hack I used.
Solution Summary
For those who prefer reading code more than reading english you can clap at this story and go to this commit. 😸
My solution leverages two things:
- Generated files from proto files are accessible in
go_generated_srcs
on the targetgo_proto_library
. So I can write a bazel target calledproto_link
which would copy the generated source into the folder where proto file was. - I can write a small Gazelle extension to add
proto_link
target to allBUILD
files where I havego_proto_library
. This will automate away the job of me placing it manually and so I will just have to runbazel run //:gazelle
to make sure whenever I add more proto code myBUILD
files will have the right contents.
Ok…I need more details
Gazelle Extension
As we talked about in golang monorepo part 1, gazelle is this super powerful magical tool that can generate bazel BUILD
files for you.
How gazelle works is outside the scope of this post but basically it has concept of a language, and go
and proto
are two languages implemented by default. How do they work ? the implementation can literally read .go
and .proto
files and figure out dependencies and generate BUILD
files for your code.
Gazelle is also extensible, you just need to implement this interface, inform bazel build system and bazel will run it when you run bazel run //:gazelle
like we were doing in the previous posts. Gazelle library provides you with bunch of helpful metadata using which you can write your implementation, here are more details and examples.
Reading go_generated_srcs
Here is the logic for copying over the generated sources.
Line 4, get the generated srcs for the go_proto_library for hello.proto
Line 13, basically string interpolation of copy_into_workspace.sh it ends up being basically the cp
command from line 7
And that is it, the whole implementation is here
Stitching Together
Now that we have written a gazelle extension which is going to place a target go_proto_link
in each BAZEL
file where there is a proto. And we have an implementation of go_proto_link
we can tell bazel build system to run it
This should be fairly obvious, we declared a gazelle_binary
on Line 4 which knows about the default languages and the one that we have written.
Line 11 is the target that gets invoked when you run bazel run //:gazelle
Demo
Run bazel run //:gazelle
Awesome, this generated the go_proto_link
target in protos/hello/BUILD.bazel
. Now you can write a script that queries for all such targets and runs them, that is easy
bazel query 'kind("proto_link", //...)' | xargs bazel run
And you are done. Whenever you now update a proto file or add a new one, you can run this script and be sure that you will have the updated auto generated code.
Use it in your project
UPDATE: You can directly use this library I wrote. Follow installation instructions there
Note
Your IDE and invoking go build
will only work if your proto files follow the same folder structure as you expect the import to be. So for example if my hello.proto
had optional go_package = "github.com/nikunjy/go/blahblahhello"
at the top of the file then this hack would not work because the script would copy it in the source folder which is go/protos/hello
thus making it accessible by import github.com/nikunjy/go/protos/hello
but the bazel build system which is how you are building and publishing images would expect the import to be github.com/nikunjy/go/blahblahhello