Jan 26, 2021

9 min read

Catching Up with the World — Go Modules in a Monorepo

November, 2020 — Go Modules (the standard for Go dependency management) had been out for over 2 years since its experimental release in go1.11, but Compass was still vendoring dependencies. Changing the dependency management tool for a monorepo isn’t an easy task, so we put it off for as long as possible. However, we were experiencing issues with an end-of-life’d vendoring tool, hundreds of megabytes of vendored dependencies, and an inability to integrate some of the latest and greatest dependencies. We couldn’t wait any longer — we needed to catch up with the world.

This is the story of that journey to move our Go dependency management off of vendoring and onto Go Modules. I’ll walk you through some of the major decisions that we made in hopes that it might help you on a similar transition.

The major decisions we navigated include: 1) Single vs Multi-Module, 2) go.mod placement, and 3) How to keep the go.mod/go.sum files stable.

1) Single vs Multi-Module:

One of the first things we needed to decide was whether or not we wanted a single module or multiple modules in our repository.

The Case for a Single Module:

Go recommends sticking to one module for one repository. Having multiple modules requires more deliberate management of the infrastructure.

For instance, a single module repository can easily import code between services and packages, unlike a multi-module repository, which will most likely want to make use of “replace directives”. Replace directives allow neighboring modules to import each others’ code without first publishing and remotely fetching it. However, they aren’t the most robust solution because they can be tricky to maintain (e.g. certain commands such as `go list` won’t work as expected) and require a decent amount of boilerplate code (e.g. replace directives are ignored outside of the current module being run/developed in, so you’ll need to repeat a replace directive in every `go.mod` in the repo where it’s relevant).

Another example of simplification with a single module is the ease of running all unit tests in the repo for continuous integration. With a single module, you can test all services/packages with a single `go test ./…`. On the other hand, a multi-module repo will require some sort of infrastructure to test each service or package (such as a script to run tests from each directory).

It is worth noting that Go will only fetch and manage dependencies for code that is directly called and being built, tested, or run, so performance issues related to fetching other services’ dependencies are NOT a concern with a single module.

The Case for Multi-Module:

According to Go documentation, a module is defined as: “A is a collection of related Go packages that are versioned together as a single unit.” Therefore, having a module per service more closely reflects the autonomy by which any single service is versioned and deployed (unless you have a single deployment that deploys all your services in the monorepo at once).

Another benefit of multiple modules is that it enables each service/package to tightly manage its dependencies. With each service managing its dependencies, you don’t have to worry about conflicting versions of dependencies. Individualized modules are more succinct and legible, making it immediately clear what dependencies a service/package has. Finally, each service’s go.mod/go.sum file(s) will change less frequently (frequent, unexpected changes to go.mod/go.sum files can be problematic, I’ll discuss it more in part 3 of this post).

Compass’s Verdict: Single Module

For the Compass backend monorepo, we decided a single module would be best because it would require the least amount of management.

We decided that, in this case, it was important to use the tool the way it was intended to be used. In past projects, we found that diverging from Go’s “1 repo = 1 module” rule was more of a headache than it was worth.

We weren’t too concerned about modules accurately reflecting individually versioned services/packages because Compass has already had to deal with this problem. Compass has a CI/CD release strategy that helps ensure that services are tested and verified so that they can be released independently without concerns of version incompatibilities (read more about it in this post).

Concerns about services/packages needing tight control over their dependencies were minimal because we found that, so far, there aren’t any dependency version conflicts in our 135+ third-party dependencies. If any conflicts happen in the future, we can use replace directives as a workaround. We also have the option to build a service outside of the monorepo if necessary.

With a single module, there remains the problem of frequent maintenance of the shared go.mod/go.sum files, which I will address in the third section of this blog post.

2) go.mod Placement:

Now that we’d decided on a single module repo, we needed to figure out where to place the go.mod file that declares the module root. Go recommends placing the go.mod file at the root of the repo, so the ideal monorepo structure would look something like this (thank you flowerinthenight and Burak Tasci!):

Ideal single module monorepo directory structure

However, we did not have that luxury. At Compass, our 3 language monorepo has a directory structure that looks something like this:

Compass pre-modules monorepo directory structure

so we could not simply place the go.mod file at the root.

Working with this structure, the first part of our solution was to put the go.mod file under the `src/go/compass.com/` directory, and pretend like that’s the root of the Go part of the repository. This is a common pattern I’ve seen in repositories that contain more than just Go code. All it requires is that you change directories to the Go module (`src/go/compass.com/` in our case) when developing Go code. The result is something like this:

We were able to do this because all our code already had import paths of the format `compass.com/pkg1`. Putting the go.mod at the root would have required changing all of those import paths, so that was part of the reason we decided to put the go.mod under `src/go/compass.com/` (not to mention the ugly module paths we would have had which would have looked like `backend_monorepo/src/go/compass.com/pkg1`).

Deciding to put the go.mod under `src/go/compass.com/` was straightforward enough. Our major complexity came from the fact that we have automation in place that calls Go commands from the root of the repository (for historical reasons that are out of the scope of this blog post). It is expected that `go test compass.com/service1/…` will work from the root of the repository.

To work around this, we also set up a “pointer” go.mod at the root that imports the `src/go/compass.com/` module (using a replace directive) and does nothing else (it is not the “true” go.mod). This enables Go commands that call code under `src/go/compass.com/` to be called from the root. So far, this strategy has worked well. However, it requires some additional care making sure that new dependencies are added to the `src/go/compass.com/` module and NOT the root module. Our repo now looks something like this:

So in truth, I technically was lying when I said we have a “single module monorepo” and if I’m being completely honest, we have a third module somewhere else in the repository, but I won’t go into that for the sake of simplicity.

3) Keeping the go.mod/go.sum Stable

As I was migrating our monorepo onto Go Modules, I found that the go.mod and go.sum files would sometimes unexpectedly update (Go automatically updates those files when appropriate). Updates would happen for different commands, sometimes a result of `go test`, sometimes `go mod tidy`, and sometimes our linter or other tooling. We knew that we wanted to keep the go.mod/go.sum files stable so that developers weren’t frequently dealing with changes to those files (which is particularly a concern with a monorepo such as ours where we’re dealing with 170+ services/packages and 100+ Go contributors).

To help keep the go.mod and go.sum files stable we have put in place the following automation:

`go mod tidy` on pre-push: The `go mod tidy` command helps keep go.mod files clean by removing unused dependencies. So we added a pre-push check that runs `go mod tidy` and blocks `git push` if any uncommitted changes are detected in the go.mod or go.sum as a result of tidying. Changes to those files must be committed before the code can be pushed up for review.
`go mod tidy` in CI: For redundancy, we run that same `go mod tidy` check again in CI (in case the pull request for which CI is running was created through some alternative means that did not run the pre-push check).
`go test -mod=readonly` in CI: When we run the suite of Go unit tests, we have the `-mod=readonly` flag on for the `go test` command. This flag causes the `go test` command to fail if it’s missing a dependency and wants to update the go.mod file. This helps ensure that dependencies are deliberately added. The go.mod must be kept up-to-date so that the next developer who pulls master doesn’t have any unexpected or unrelated go.mod changes.
`go.mod` updates pushed up for review in a cron job: I mentioned before that we would unexpectedly get updates to the go.mod files. To help combat this we have set up a cron job that runs a suite of Go commands (such as `go test` and `go mod tidy`, as well as linting and other tooling). If any go.mod or go.sum changes are found after running these commands, then a pull request containing those changes is generated and pushed up for review. At the moment, we’re running it once a day.

Summary:

First, we decided whether we wanted a single module or multiple modules. A single module is the Go default, but multiple modules might help with versioning and will give each service more control. We chose the single module approach, given Go’s “1 repo = 1 module” ideal.
Next, we decided where the go.mod file should live. The easiest location is the root of the repository (the Go default), but we didn’t have that flexibility. So we place the go.mod at the root of the Go code (with a go.mod at the root, purely to act as a pointer so we could run Go commands from the root).
Finally, we keep our go.mod/go.sum files stable by having checks for `go mod tidy`, run tests in CI in `-mod=readonly` mode, as well as automation that looks for go.mod updates and pushes them up for review.

I hope that by sharing our journey, I’ve given you some food for thought as you set up your own Go Modules monorepo. Please comment and share your experience building monorepos with Go Modules!

Catching Up with the World — Go Modules in a Monorepo

1) Single vs Multi-Module:

2) go.mod Placement:

3) Keeping the go.mod/go.sum Stable

Summary:

Related Links:

More from Compass True North

Get the Medium app

Nathaniel Morihara