Golang with Bazel

In this short write-up, we will go over how to use Golang with the Bazel build system. In particular, we will go over three scenarios: Starting a Golang project from scratch, converting an existing Golang project to Bazel and pulling in an external Golang project into your Bazel build system.

Starting Golang project from scratch

Let us start with the basics of just using Go with Bazel. For that, we need to pull in the official rules for the Go language from https://github.com/bazelbuild/rules_go.

In the setup section, you will find the following piece of Starlark code that we need to put into a file named WORKSPACE.

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "io_bazel_rules_go",
    sha256 = "8e968b5fcea1d2d64071872b12737bbb5514524ee5f0a4f54f5920266c261acb",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.28.0/rules_go-v0.28.0.zip",
        "https://github.com/bazelbuild/rules_go/releases/download/v0.28.0/rules_go-v0.28.0.zip",
    ],
)

load("@io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies")

go_rules_dependencies()

go_register_toolchains(version = "1.16.5")

Let us step through what is happening here. First, we use the load directives to pull new features for use in our Bazel files. We do this twice to import the ability to download HTTP repositories and then load Go-specific commands from the now downloaded repository.

For the import itself, we need to provide a name. The usual naming scheme is the reverse domain, followed by the namespace and project; all converted into underscores. E.g. github.com/user/projectbecomes com_github_user_project. This project is part of the official Bazel organization, so it is using the bazel.io domain instead of github.com.

If you are not familiar with Bazel, the actual build configuration is done through BUILD files, where we can treat Go as any other language and use rules following the same structure: go_binary, go_library and go_test. I have prepared a minimal example on my Github: https://github.com/HappyCerberus/bazel-golang-minimal-example. You will notice that we need to load these rules from the imported io_bazel_rules_go repository to make them available in the BUILD file.

Converting an existing project into Bazel

So, starting from scratch is easy, but what if you already have a Golang project and you need to convert it into Bazel? For that, we need to use another tool available in the official Bazel project called Gazelle (https://github.com/bazelbuild/bazel-gazelle).

For demonstration, I will use a third-party project (https://github.com/aler9/rtsp-simple-server) that I’m currently modifying for my upcoming system design course (follow me to get notified when it launches).

First, we need to create a WORKSPACE file and copy-paste the code from the setup section of the Gazelle repository.

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "io_bazel_rules_go",
    sha256 = "8e968b5fcea1d2d64071872b12737bbb5514524ee5f0a4f54f5920266c261acb",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.28.0/rules_go-v0.28.0.zip",
        "https://github.com/bazelbuild/rules_go/releases/download/v0.28.0/rules_go-v0.28.0.zip",
    ],
)

http_archive(
    name = "bazel_gazelle",
    sha256 = "62ca106be173579c0a167deb23358fdfe71ffa1e4cfdddf5582af26520f1c66f",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/bazel-gazelle/releases/download/v0.23.0/bazel-gazelle-v0.23.0.tar.gz",
        "https://github.com/bazelbuild/bazel-gazelle/releases/download/v0.23.0/bazel-gazelle-v0.23.0.tar.gz",
    ],
)

load("@io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies")
load("@bazel_gazelle//:deps.bzl", "gazelle_dependencies")

go_rules_dependencies()

go_register_toolchains(version = "1.16.5")

gazelle_dependencies()

You will notice that this is also importing rules mentioned in the previous section.

Now, to actually run Gazelle, we need to add it to our main BUILD file.

load("@bazel_gazelle//:def.bzl", "gazelle")

# gazelle:prefix github.com/aler9/rtsp-simple-server 
gazelle(name = "gazelle")

The prefix specification refers to the Go import path used across the project. For example the main.go has the following imports:

import (
    "os"
    "github.com/aler9/rtsp-simple-server/internal/core"
)

At this point, we can actually run Gazelle and let it generate BUILD files for our project.

$ bazel run //:gazelle

After which, we should have all the BUILD files for the project automatically generated:

$ git status
On branch main
Your branch is up to date with 'origin/main'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)
 BUILD
 WORKSPACE
 bazel-bin
 bazel-out
 bazel-test
 bazel-testlogs
 internal/aac/BUILD.bazel
 internal/conf/BUILD.bazel
 internal/confenv/BUILD.bazel
 internal/confwatcher/BUILD.bazel
 internal/core/BUILD.bazel
 internal/externalcmd/BUILD.bazel
 internal/h264/BUILD.bazel
 internal/hls/BUILD.bazel
 internal/logger/BUILD.bazel
 internal/rlimit/BUILD.bazel
 internal/rtcpsenderset/BUILD.bazel
 internal/rtmp/BUILD.bazel
nothing added to commit but untracked files present (use "git add" to track)

However, if you try to build the project using bazel build //... you will actually see many errors concerning undefined repositories. This is because we are still missing the definitions of the project dependencies. However, Gazelle can do that for us as well (to_macro part is optional):

$ bazel run //:gazelle -- update-repos -from_file=go.mod -to_macro=deps.bzl%go_dependencies

This command will generate a new deps.bzl file (or edit WORKSPACE if we omit the to_macro directive) and transitively import all repositories that we need to build the project.

$ cat deps.bzl
load("@bazel_gazelle//:deps.bzl", "go_repository")
def go_dependencies():
    go_repository(
        name = "com_github_alecthomas_template",
        importpath = "github.com/alecthomas/template",
        sum = "h1:JYp7IbQjafoB+tBA3gMyHYHrpOtNuDiK/uB5uXxq5wM=",
        version = "v0.0.0-20190718012654-fb15b899a751",
    )
    go_repository(
        name = "com_github_alecthomas_units",
        importpath = "github.com/alecthomas/units",
        sum = "h1:UQZhZ2O0vMHr2cI+DC1Mbh0TJxzA3RcLoMsFw+aXw7E=",
        version = "v0.0.0-20190924025748-f65c72e2690d",
    )
    go_repository(
        name = "com_github_aler9_gortsplib",
        importpath = "github.com/aler9/gortsplib",
        sum = "h1:Bf0hzdN1jUWsb5Mzezq1pd18EIBeKXxk5clIpHZJ1Lk=",
        version = "v0.0.0-20210724151831-dae5a1f04033",
    )
    go_repository(
...

In the case of this repository, I actually run into a small hitch here. The build still failed due to the import of org_golang_x_tools which was incorrectly deduced as a dependency (fixed by removing it from deps.bzl). You can see the final result at https://github.com/HappyCerberus/rtsp-simple-server (my fork of the rtsp-simple-server project).

You can continue using Gazelle to manage the dependencies, which is also how you can pull in a repository into your Bazel based project without actually converting it.

$ bazel run //:gazelle -- update-repos github.com/some/repo

Hermetic tests

One last problem that you will likely run into is hermetic tests. If you see your tests fail with access denied, file not found or operation not permitted failures, it is because Bazel enforces hermetic tests. This means that each test must be fully self-contained and independent of any other test.

For unit tests, any files need to be provided as dependencies to the test and accessed through the runfiles mechanism (https://github.com/bazelbuild/rules_go/blob/master/go/tools/bazel/runfiles.go).

A temporary directory for each test is provided in the TEST_TMPDIR environmental variable instead of the typical os.TempDir().

Hermetic integration and system tests require careful design from the start, so converting existing tests of this type might be tricky. Sadly I do not have a one-size-fits-all recommendation here.

While converting your tests to be hermetic can be annoying, it is a worthwhile effort that will bring your better test repeatability and lower flakiness.

Thank you for reading

Thank you for reading this article. Did you enjoy it?

I also publish videos on YouTube: youtube.com/c/simontoth and if you want to chat, hit me up on Twitter @SimonToth83 or LinkedIn linkedin.com/in/simontoth.