How to Golang Monorepo

Published in

goc0de

6 min readMay 12, 2020

Some weeks ago, I was trying to setup golang in a monorepo and I could not really find a definitive guide on how to do it easily. I have linked some articles in the references below that were useful to me, but I am hoping this article series is a definitive guide and has all the information about how to do your golang monorepo the “right” way, relevant in 2020.

Guiding principles behind the choices in this article are:

Easy to write code in monorepo keeping dev xp in mind. Editor works, dependency management is easy, documentation on the tools exist etc.
Easy to build and test locally as well as through github actions, or buildkite etc.
Builds are cached and fast.

To this end, we are going to be making use of (I strongly urge you consider these) :

Go modules cause they make it easy to manage dependencies. Vim-go, VsCode, and Goland all know how to work with go modules.
Bazel because it is a build tool that is oblivious to the languages you are using in your monorepo and it is very easily extensible.
Use bazel to produce docker images

In the interest of not making this article too long to read, I will split it and in the second one explore:

Making the docker setup more customized.
GRPC + Protocol Buffers because in all probability you will end up using them eventually.

Go Modules

Go modules are now the default way of doing dependency management in golang. We have finally arrived and we no longer need to use other tools like dep, Godep, gvt, or glide. I feel the need to highlight it especially for folks who have written golang on and off in the past.

Let me assume my monorepo is called go and we want to locate it in the GitHub repo github.com/nikunjy/go. In this mono repo I want all the services under the folder services and libraries under libraries and I want all the internal imports to be github.com/nikunjy/go/<something>.

Let’s get started

> mkdir -p go; cd go> go mod init github.com/nikunjy/go

This creates a file go.mod that has the following:

module github.com/nikunjy/gogo 1.14

This file simply informs that a module named github.com/nikunjy/go lives here, all the dependencies we use will be listed ultimately in this file. Now I write my code with a main.go and a server.go and my repo looks like this

Couple of code snippets to note here are:

Hello http handler in server.go

Note the prefix to the server pacakage is github.com/nikunjy/go

Bazel

Bazel is a pretty awesome build tool that I would encourage you to learn about. Here are some things to note about bazel:

It can build anything, it doesn’t care what language your code is written in.
You can easily write bazel files using python like looking code
Bunch of talented folks have written bazel modules that you can build your stuff on top of.

Quick summary about bazel

WORKSPACE defines a workspace much like how how you open a project in an IDE. It contains all the bazel imports and incantations for the BUILD.bazel or BUILD files to make sense.
BUILD.bazel are equivalent Makefile for bazel system. There is probably going to be one in each subfolder of your repo
Build files have two kinds of syntaxes load() and a macro.
Load is to import some other bazel module (you might have written one or you are pointing to someone’s). This works because you informed where this module lives in WORKSPACE file by using git_repository or http_archive
Macros are function calls that you have imported using load and now you can call them in your build files.

Build golang with Bazel

To build golang code with bazel, all you need is the rules_go and gazelle module. I wasted a lot of time looking for how to build proto code in golang, and I actually didn’t need anything except those two modules, so trust me when I say you probably won’t need anything else for the time being.

rules_go provides bazel macros to build go code, two that you will get familiar with are go_library and go_binary . They are exactly what they sound like, we will look at these in a minute.

gazelle generates BUILD.bazel files for your entire go code contained in monorepo. BUILD.bazel files are like Makefile for bazel system

So first install gazelle using instructions, after this step gazelle should be a recognized command

go get github.com/bazelbuild/bazel-gazelle/cmd/gazelle➜  go git:(master) gazelle
gazelle: -repo_root not specified, and WORKSPACE cannot be found: file does not exist# YAY!

Make a file WORKSPACE and make its contents same as this link:

Make a file called BUILD and make its contents as

Note the #gazelle:prefix github.com/nikunjy/go tells gazelle that we are operating in a golang module and import path starts with the prefix github.com/nikunjy/go which is what we want as I tried to get your attention in the main.go snippet.

Now run the command bazel run //:gazelle which tells bazel to run the gazelle target specified in the BUILD file. This will autogenerate the BUILD.bazel files for all of the packages.

Let’s inspect what the repo looks like at this point

➜  go git:(master) ✗ git status
On branch master
Your branch is up to date with 'origin/master'.Untracked files:
  (use "git add <file>..." to include in what will be committed)
 BUILD
 WORKSPACE
 bazel-bin 
 bazel-go
 bazel-out
 bazel-testlogs
 services/hello/BUILD.bazel
 services/hello/server/BUILD.bazel

Four bazel-* folders are where bazel compiles and stores state for this project. These will be ignored when I do git commit
services/hello/BUILD.bazel is the bazel equivalent Makefile for the package services/hello
Ditto for services/hello/server/BUILD.bazel

Let’s inspect one of the .bazel files, I am inlining the comments

Okay, git add; git commit

Now, let’s inform bazel about the dependencies mentioned in go.mod file because if bazel doesn’t know about them how will it compile your project ?

This will produce a file go_third_party.bzl which will have the “bazelified” representation of your go.mod files. It will also inform WORKSPACE aware of this file:

> git diff WORKSPACE

+load("//:go_third_party.bzl", "go_deps")
+
+# gazelle:repository_macro go_third_party.bzl%go_deps
+go_deps()

We are DONE ! Lets try to compile and run my awesome hello service

Compile: bazel build //services/hello

Run: bazel run //services/hello

Bah ! It is running on port 0, let me pass an argument I wrote in my code proxy-port

Build docker images using bazel

I am gonna be using rules_docker to build docker images for my hello service. Not only that, rules_docker can push the container image to ECR or GCR thus making your docker image available in your image repository by running commands locally or through your CI/CD system.

You can look at this git commit to follow along

First change the workspace to add rules_docker

Now Add following lines toservices/hello/BUILD.bazel

Some notes here:

go_image defines that a binary will be built with go_default_library (this target was already generated by gazelle in previous steps)
go_image accepts bunch of arguments and by default this image is built on top of distroless. Distroless is supposed to be a secure base to build on top of, here is a good article explaining it briefly.
container_image is building essentially a docker image based on the go_image. It accepts bunch of arguments including passing environment variables and such.

Let’s test this out shall we ? Lets run the target //services/hello:image which is the container_image target

go git:(master) bazel run //services/hello:image
INFO: Analyzed target //services/hello:image (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //services/hello:image up-to-date:
  bazel-bin/services/hello/image-layer.tar
INFO: Elapsed time: 0.271s, Critical Path: 0.01s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
Loaded image ID: sha256:c28e3dc782622f03bb0ca4e6517ce0f2230edcc26bcab7a01a1ad75065fbe546
Tagging c28e3dc782622f03bb0ca4e6517ce0f2230edcc26bcab7a01a1ad75065fbe546 as bazel/services/hello:image

That command produces a docker image and it will show on your local docker

➜  go git:(master) docker images
REPOSITORY                             TAG                 IMAGE ID            CREATED             SIZE
bazel/services/hello                   image               c28e3dc78262        24 hours ago        24.4MB

That is all I have to share in this post, I hope you found it useful. If you did then please let me know in the comments or clap. In the next post we will talk more about the docker setup and grpc + protobuf.

How to Golang Monorepo

Go Modules

Bazel

Build docker images using bazel

References

Written by Nikunj Yadav