The battle for an IDE-friendly stack trace in Go (with and without Bazel)

Author: Artem Navrotskiy

Developing software is about more than just writing code - you have to debug as well. So why not make debugging as painless as possible?

With some errors, we write the stack trace to the log. The IDE we use (Idea, GoLand) allows for convenient file navigation using a copied call stack (Analyze external stack traces). Unfortunately, this feature only works well if the binary is built on the same host that the IDE is running on.

This post is about how we tried to create synergy between the call stack format and the IDE.

What kind of stack display options does Go Build provide?

There are two controls in Go Build for influencing the stack output format:

• the -trimpath flag: causes the display of the call stack to be the same, regardless of the local location of the files;
• the GOROOT_FINAL environment variable: lets you replace the prefix to system libraries on a stack when the -trimpath flag is off.

A program for comparing stack trace output

Let’s consider stack mapping using the example of a small program. The source code can be downloaded here, and here’s the program (stacktrace/main.go):

package main

import (
	"fmt"

	"github.com/Masterminds/cookoo"
	"github.com/pkg/errors"
)

func main() {
	// Build a new Cookoo app.
	registry, router, context := cookoo.Cookoo()
	// Fill the registry.
	registry.AddRoutes(
		cookoo.Route{
			Name: "TEST",
			Help: "A test route",
			Does: cookoo.Tasks{
				cookoo.Cmd{
					Name: "hi",
					Fn:   HelloWorld,
				},
			},
		},
	)
	// Execute the route.
	router.HandleRequest("TEST", context, false)
}

func HelloWorld(cxt cookoo.Context, params *cookoo.Params) (interface{}, cookoo.Interrupt) {
	fmt.Printf("%+v\\n", errors.New("Hello World"))
	return true, nil
}

And then let’s add a little go.mod:

module github.com/bozaro/go-stack-trace

go 1.20

require (
	github.com/Masterminds/cookoo v1.3.0
	github.com/pkg/errors v0.9.1
)

The tried and true GOPATH

To keep things in perspective, let's start with some good old GOPATH.

Sample output:

➜ GO111MODULE=off GOPATH=$(pwd) go get -d github.com/bozaro/go-stack-trace/stacktrace
➜ GO111MODULE=off GOPATH=$(pwd) go run github.com/bozaro/go-stack-trace/stacktrace 
Hello World
main.HelloWorld
	/home/bozaro/gopath/src/github.com/bozaro/go-stack-trace/stacktrace/main.go:31
github.com/Masterminds/cookoo.(*Router).doCommand
	/home/bozaro/gopath/src/github.com/Masterminds/cookoo/router.go:209
github.com/Masterminds/cookoo.(*Router).runRoute
	/home/bozaro/gopath/src/github.com/Masterminds/cookoo/router.go:164
github.com/Masterminds/cookoo.(*Router).HandleRequest
	/home/bozaro/gopath/src/github.com/Masterminds/cookoo/router.go:131
main.main
	/home/bozaro/gopath/src/github.com/bozaro/go-stack-trace/stacktrace/main.go:27
runtime.main
	/usr/lib/go-1.20/src/runtime/proc.go:250
runtime.goexit
	/usr/lib/go-1.20/src/runtime/asm_amd64.s:1598

Everything here is straightforward - we see the full paths to each file. In this case, all paths are located either in the src directory of GOROOT or in the GOPATH directory.

Unfortunately, such a stack only points to existing files if the executable is built in an environment with the same directory layout. In our case, where some developers use MacOS, and the product build environment uses Linux, this requirement is not feasible.

Luckily, there’s a -trimpath flag that removes the troublesome part of the call stack:

➜ GO111MODULE=off GOPATH=$(pwd) go run -trimpath github.com/bozaro/go-stack-trace/stacktrace
Hello World
main.HelloWorld
	github.com/bozaro/go-stack-trace/stacktrace/main.go:31
github.com/Masterminds/cookoo.(*Router).doCommand
	github.com/Masterminds/cookoo/router.go:209
github.com/Masterminds/cookoo.(*Router).runRoute
	github.com/Masterminds/cookoo/router.go:164
github.com/Masterminds/cookoo.(*Router).HandleRequest
	github.com/Masterminds/cookoo/router.go:131
main.main
	github.com/bozaro/go-stack-trace/stacktrace/main.go:27
runtime.main
	runtime/proc.go:250
runtime.goexit
	runtime/asm_amd64.s:1598

In this case, all paths will relative to either GOPATH or src in the GOROOT directory, and it ended up being a completely portable call stack format.

Go Modules

When using Go Modules, the behavior of the -trimpath flag changes dramatically. Let’s compare the output of the call stack without it:

➜ git clone <https://github.com/bozaro/go-stack-trace.git> .
➜ go run ./stacktrace 
Hello World
main.HelloWorld
	/home/bozaro/github/go-stack-trace/stacktrace/main.go:31
github.com/Masterminds/cookoo.(*Router).doCommand
	/home/bozaro/go/pkg/mod/github.com/!masterminds/cookoo@v1.3.0/router.go:209
github.com/Masterminds/cookoo.(*Router).runRoute
	/home/bozaro/go/pkg/mod/github.com/!masterminds/cookoo@v1.3.0/router.go:164
github.com/Masterminds/cookoo.(*Router).HandleRequest
	/home/bozaro/go/pkg/mod/github.com/!masterminds/cookoo@v1.3.0/router.go:131
main.main
	/home/bozaro/github/go-stack-trace/stacktrace/main.go:27
runtime.main
	/usr/lib/go-1.20/src/runtime/proc.go:250
runtime.goexit
	/usr/lib/go-1.20/src/runtime/asm_amd64.s:1598

And a similar output with -trimpath:

➜ go run -trimpath ./stacktrace
Hello World
main.HelloWorld
	github.com/bozaro/go-stack-trace/stacktrace/main.go:31
github.com/Masterminds/cookoo.(*Router).doCommand
	github.com/Masterminds/cookoo@v1.3.0/router.go:209
github.com/Masterminds/cookoo.(*Router).runRoute
	github.com/Masterminds/cookoo@v1.3.0/router.go:164
github.com/Masterminds/cookoo.(*Router).HandleRequest
	github.com/Masterminds/cookoo@v1.3.0/router.go:131
main.main
	github.com/bozaro/go-stack-trace/stacktrace/main.go:27
runtime.main
	runtime/proc.go:250
runtime.goexit
	runtime/asm_amd64.s:1598

Without -trimpath we still see full paths to each file, while clearly tracing three types of source files:

• a working copy directory (in this example: $HOME/github/go-stack-trace);
• GOROOT system libraries from $GOROOT/src (in this example: /usr/lib/go-1.20/src);
• third-party libraries from $GOMODCACHE (in this example: $HOME/go/pkg/mod);

At the same time, unlike GOPATH, the -trimpath flag does not cut off the prefix in file names:

• files from the current module in the working copy directory are named with the module name from go.mod as a prefix (in this example: $HOME/github/go-stack-trace → github.com/bozaro/go-stack-trace);
• GOROOT system libraries from $GOROOT/src get filenames without a prefix;
• third-party libraries get the module name with the version as a prefix (in this example: /home/bozaro/go/pkg/mod/github.com/!masterminds/cookoo@v1.3.0 → github.com/Masterminds/cookoo@v1.3.0, and it’s worth noting that the word Masterminds in the file path and module name is spelled differently).

Which stack trace is IDE-friendly?

If you happen to open a project from the repository in Idea / GOROOT and try to analyse any of the above call stacks, then there won’t be any navigation through the source files:

• call stack options for GOPATH aren’t suitable because this mini-project uses Go Modules and has a different file layout;
• the option for Go Modules without -trimpath won't work because your home directory will most likely be different from /home/bozaro;
• the option for Go Modules with -trimpath won't work as it’s not supported in IDE (https://youtrack.jetbrains.com/issue/GO-13827), and of all the paths that are visible on the stack, only files from the Go SDK will be the suffixes of existing files.

It seems like the IDE in our case is looking for source files along the paths relative to the project directory and its parents. As a result, an acceptable format for a portable call stack is as follows:

main.HelloWorld
	stacktrace/main.go:31
github.com/Masterminds/cookoo.(*Router).doCommand
	go/pkg/mod/github.com/!masterminds/cookoo@v1.3.0/router.go:209
github.com/Masterminds/cookoo.(*Router).runRoute
	go/pkg/mod/github.com/!masterminds/cookoo@v1.3.0/router.go:164
github.com/Masterminds/cookoo.(*Router).HandleRequest
	go/pkg/mod/github.com/!masterminds/cookoo@v1.3.0/router.go:131
main.main
	stacktrace/main.go:27
runtime.main
	GOROOT/src/runtime/proc.go:250
runtime.goexit
	GOROOT/src/runtime/asm_amd64.s:1598

As a result:

• paths to project files are displayed relative to the project root;
• as paths to third-party dependencies, the path to the module relative to $GOMODCACHE is used, but with the prefix go/pkg/mod (IDE will find this path when the project is in the home directory, and the environment variables GOPATH and GOMODCACHE have a default value);
• we use GOROOT as Go SDK file prefix. This marker allows you to visually identify these files, but we failed to get the IDE to provide navigation on them.

With this call stack format, the IDE recognizes all files except those from the Go SDK. The whole thing breaks if a developer locally overrides the GOPATH or GOMODCACHE environment variables, but there’s generally no need to do this.

Getting the call stack in the right format

This can be done using the following methods:

• modify debug infromation in compile time;
• before output, convert the call stack to the desired format;
• make an external utility that converts the call stack to the required format.

Modify debug information in compile time

With Go Build, we cannot modify debug information in compile time to get the desired format of the stack trace.

Stack conversion before output

In our case, we use the library github.com/joomcode/errorx across the board, and it has a method for converting the call stack to the desired format before output (you can find that here).

Converting a path from a view without -trimpath is trivial, but this method has a number of disadvantages:

• if the call stack is passed by this filter, then it will remain in the original format;
• some places, like pprof, are guaranteed to be transmitted in their original format.

External utility

Using an external utility greatly complicates the overall call stack parsing scenario. In our case (and in most cases), the stack was taken from the logs where it was already in the necessary format, so we didn’t seriously consider this option.

Migration to Bazel forced me to tackle this problem again

In general, converting a stack before log write was suitable to the Bazel build migration. But building by Bazel brought the problem to a new level.

Stack from the binary built by Bazel

➜ bazel run //stacktrace 
...
INFO: Running command line: bazel-bin/stacktrace/stacktrace_/stacktrace
Hello World
main.HelloWorld
	stacktrace/main.go:31
github.com/Masterminds/cookoo.(*Router).doCommand
	external/com_github_masterminds_cookoo/router.go:209
github.com/Masterminds/cookoo.(*Router).runRoute
	external/com_github_masterminds_cookoo/router.go:164
github.com/Masterminds/cookoo.(*Router).HandleRequest
	external/com_github_masterminds_cookoo/router.go:131
main.main
	stacktrace/main.go:27
runtime.main
	GOROOT/src/runtime/proc.go:250
runtime.goexit
	src/runtime/asm_amd64.s:1598

We don't require developers to to use Bazel to build and run tests for a number of reasons, some of which are:

• we generate BUILD files with our utility and don’t want to require regeneration of files for every change (it’s fast, but not instantaneous);
• Synchronization of IDE and BUILD files is rather slow.

At the same time, in the call stack from Bazel:

• third-party libraries begin to refer to the external directory, which the IDE doesn’t see;
• it’s impossible to get the path to the module in GOMODCACHE in a straightforward way - information about the module version is lost;
• generated files can get a completely unexpected prefix like bazel-out/k8-fastbuild-ST-2df1151a1acb/....

All of these paths refer to real files and are quite meaningful in the context of Bazel, but without full integration they only complicate the matter.

Stack conversion before output

Initially, they tried to assemble a set of rules that allow you to form something acceptable from the existing call stack.

To do this (through x_defs and then embed ), a separately generated file was passed to the program, which contained the correspondence of the external name to the desired prefix in the call stack.

We also made a number of transformations to process the paths of generated files. The problem became less acute, but the results were still unsatisfactory:

• pprof became a complete nightmare;
• some of the paths were converted incorrectly;
• the structure as a whole was quite complex and fragile.

External utility

I didn’t want to have to go down this path: in addition to all the complexity and fragility when converting the stack before output, there was also the problem of putting information to this utility that we integrated into the executable file, namely, matching the external name to the desired prefix in the call stack.

That is, it seemed necessary to do a stack trace deobfuscator. The problem is that we would obfuscate the code only from ourselves.

Modify debug information in compile time

When using Bazel, the build is at a lower level than Go Build. We had hoped to fix the assembly in order to immediately have convenient paths to files. The $(go env GOTOOLDIR)/compile utility also has a -trimpath option, but this parameter is no longer a boolean flag, but rather a list for replacing prefixes.

As a result, we added additional attributes to the go_library and go_repository rules so that we could set stack trace prefix for the library:

• https://github.com/bazelbuild/rules_go/pull/3307
• https://github.com/bazelbuild/bazel-gazelle/pull/1379

After these changes, you can override the path of files in the call stack, for example:

diff --git a/deps.bzl b/deps.bzl
index ffe4981..d917282 100644
--- a/deps.bzl
+++ b/deps.bzl
@@ -5,6 +5,7 @@ def go_dependencies():
         name = "com_github_masterminds_cookoo",
         importpath = "github.com/Masterminds/cookoo",
         sum = "h1:zwplWkfGEd4NxiL0iZHh5Jh1o25SUJTKWLfv2FkXh6o=",
+        stackpath = "go/pkg/mod/github.com/!masterminds/cookoo@v1.3.0",
         version = "v1.3.0",
     )
     go_repository(
@@ -12,4 +13,5 @@ def go_dependencies():
         importpath = "github.com/pkg/errors",
         sum = "h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=",
         version = "v0.9.1",
+        stackpath = "go/pkg/mod/github.com/pkg/errors@v0.9.1",
     )

A sample output in bazel branch:

➜ git checkout bazel
➜ bazel run //stacktrace
INFO: Running command line: bazel-bin/stacktrace/stacktrace_/stacktrace
Hello World
main.HelloWorld
	stacktrace/main.go:31
github.com/Masterminds/cookoo.(*Router).doCommand
	go/pkg/mod/github.com/!masterminds/cookoo@v1.3.0/router.go:209
github.com/Masterminds/cookoo.(*Router).runRoute
	go/pkg/mod/github.com/!masterminds/cookoo@v1.3.0/router.go:164
github.com/Masterminds/cookoo.(*Router).HandleRequest
	go/pkg/mod/github.com/!masterminds/cookoo@v1.3.0/router.go:131
main.main
	stacktrace/main.go:27
runtime.main
	GOROOT/src/runtime/proc.go:250
runtime.goexit
	src/runtime/asm_amd64.s:1598

NOTE: For some reason, the Gazelle patch is not picked up by itself. If an error like flag provided but not defined: -stack_path_prefix occurs while running the example, then to fix it, you need to rebuild Gazelle itself. In this case, the easiest way to reset the Bazel cache is: bazel clean --expunge && bazel shutdown.

Conclusion

As a result, we were able to ensure that the stack trace copied from the log was correctly recognized in the IDE. At the same time, this extended to all sources of stack traces, including, for example, prof full goroutine stack dump.

To do this, we had to patch up rules_go and gazelle a little, but we hope that these changes will someday get into the upstream.

Any comments, suggestions and words of support (or condemnation) are welcome, and stories about your own experience are more than welcome!