Versioned Go and the future of package management

Published in

Blue Harvest Tech Blog

8 min readNov 25, 2019

A gopher moving packages around — Picture by Ashley McNamara CC BY-NC-SA 4.0

Once upon a time

Go packaging and distribution has been always a problem. Since the earliest versions, Go packaging relies on a directory in the machine where all the code is placed. This directory, stored in a global variable named $GOPATH, it doesn’t only have your source code, but also all the dependencies it uses. Dependencies are added and pulled from control version systems like git using the command go get, and stored for all the projects.

This has several issues: you pull only one global version of the dependency which is shared by all the projects, and because it uses the URL of the repository as an identifier, you cannot have different versions of the same dependency. This version is also the last version available.

Different projects using the same dependency, but different versions, could not coexist in the same machine without some $GOPATH shenanigans. It also forces you to follow the code organization proposed by Go, which is trivial itself but can make many people feel unhappy about this.

The Go vendoring experiment

In response to the community complaints, on June 19th, 2015, the Go team added a flag to the Go toolchain named GO15VENDOREXPERIMENT which enabled vendoring dependencies in Go projects.

With the flag enabled and a vendor folder containing dependencies, it allowed to resolve the external dependencies using the vendor folder. For example, if you reference in your code to import github.com/foo/bar it will look for vendor/github.com/foo/bar in your file system. This experiment did not include any version resolution, version file, or even versioning tool. It was up to the community to fill the gap. The go get tool was still working as before.

The community quickly reacted creating tools like godep, glide or gb. Each tool had its own version resolution method, versions file, and workflow. This created chaos and discomfort in the community as there was no “official” tool and it was not clear the support and integrations for these experiments.

Finally, the Go team proposed an official tool named dep to deal with this instability feeling. The tool contained a versioning file, algorithm resolution, and workflow. It became also the “official experiment” with Go version 1.11.

`vGo` and the future of Go package management

dep was not the perfect tool, but it did the job. It got support and traction on the community until Russ Cox started his series of posts about versioned Go. In these posts, he described the evolution of the package management in Go, and how dep was one more step before the integration on the standard Go tooling. He described a system that integrates the versioning in the standard Go tooling and how to resolve dependencies using that system. He also implemented a proof of concept that is the base of the current implementation. As a novelty, this system does not make use of the GOPATH variable (only for the local cache for the modules, set to $GOPATH/pkg/mod) so the go get command retrieves the dependencies into the project’s vendor instead of the $GOPATH.

The Go team quickly adopted his proposal and ended the other experiments, claiming that vGo is the way to go. This decision created a lot of controversy in the community due to the almost unilateral decision and the position of Russ Cox as a core developer of Go.

The versioned Go is currently implemented in the standard Go tooling behind the flag GO111MODULEsince version 1.11 of Go and it is expected to be enabled by default at some point in the future. The flag has 3 possible values: on which enables the usage of the modules and it will not consult the $GOPATH looking for dependencies, off which disables the usage of modules and uses the traditional modules, and auto which is the default value and it will use the modules only if the folder is not in the $GOPATH. Modules are still considered on development and there could be issues with its usage.

One problem that it does not solve is what to do if the dependency itself disappears. It does not necessarily need to be deleted, but it can also be moved to a different location. For instance, given Github repositories, if the owner of the repository decides to create a project for it (we are not even talking about transferring the project to some potential rogue actor), it will make all the projects depending on it fail.

This problem has been solved recently in version 1.13 introducing the flags GOPROXY and GONOPROXY. This allows us to use a proxy (or not) to cache the packages so its sudden disappearance does not affect development. It defaults to https://proxy.golang.org,direct . This means that it will connect to the Google cache first and try to download the dependency from there, with fallback on a direct connection to the original URL. Other possible values are off (no proxy), or a list of comma-separated URL in order of preference, being the keyword direct the one to refer to the original URL.

How to start using Go modules

The first step is to make sure that we have one of the latest versions of Go installed in our machine. The output of go version should be at least 1.11 or higher. In this example, I will be using Go 1.13, the last currently available version.

The second step is to make sure that we have our environment variables correctly set.

export GO111MODULE=on
export GOPATH=$HOME/.go  # Optional

The workflow using go modules relies on the following commands:

go mod init — Create a new module or migrate an existing one.
go get — Add or update dependencies.
go list -m all — List the dependencies of the modules.
go mod tidy — Synchronize your dependencies with your code. It will add missing dependencies or remove the ones that are not needed anymore.
go mod vendor — Move your dependencies to a vendor folder.

Example

In a path outside our $GOPATH, we create a new folder that we will use for our example.

$ mkdir modexample
$ cd modexample
$ git init  # optional
$ go mod init github.com/threkk/how-to-use-go-modules
go: creating new go.mod: module github.com/threkk/how-to-use-go-modules

This will create a go.mod file which marks the current folder as a module. The parameter does not need to be an URL, but this is the identifier we will use to retrieve the module later on using go get Therefore, it needs to be something that can be resolved as a Go package.

Let’s create a simple module that makes use of an external package. This module will be a wrapper around ProtonMail’s OpenPGP package and it will provide an interface to encrypt and decrypt text using a PGP key.

crypto.go // github.com/how-to-use-go-modules/modexample

package modexampleimport (
    "io/ioutil""github.com/ProtonMail/gopenpgp/helper"
)type GPGKey struct {
    PubKeyPath  string
    PrivKeyPath string
    pubKey      string
    privKey     string
    Pass        string
}func (gpg *GPGKey) Encrypt(msg string) (string, error) {
    if gpg.pubKey == "" {
        buff, err := ioutil.ReadFile(gpg.PubKeyPath)
        if err != nil {
            return "", err
        }
        gpg.pubKey = string(buff)
    }
    return helper.EncryptMessageArmored(gpg.pubKey, msg)
}func (gpg *GPGKey) Decrypt(msg string) (string, error) {
    if gpg.privKey == "" {
        buff, err := ioutil.ReadFile(gpg.PrivKeyPath)
        if err != nil {
            return "", err
        }gpg.privKey = string(buff)
    }
    return helper.DecryptMessageArmored(gpg.privKey, gpg.Pass, msg)
}

So, we need to get the package. So far, the package only has one version, v1.0 which can be found in the releases page on Github. However, we want to get the latest version available, so we ask for the version in master .

$ go get github.com/ProtonMail/gopenpgp@master 
go: finding github.com/ProtonMail/gopenpgp master

Let’s try to generate a local vendor folder. This folder contains a local copy of the dependencies which will be used instead of the global cache located in the $GOPATH .

$ go mod vendor
go: finding golang.org/x/crypto latest
github.com/threkk/how-to-use-go-modules imports
github.com/ProtonMail/gopenpgp/crypto imports
golang.org/x/crypto/rsa: module golang.org/x/crypto@latest (v0.0.0-20191011191535-87dc89f01550) found, but does not contain package golang.org/x/crypto/rsa

Oops, we got an error. This error would have happened also if we tried to use any other command like go build or go test which would execute our code. All these commands that attempt to execute the code will resolve the dependencies and download them in case they are not present.

The error is precisely an error that Go modules try to solve: one of your packages does not exist anymore. In this case, it is a dependency of our dependency. To solve this, we can replace the package with a custom version of it. This is what the original package does, but it does not propagate to our go.mod file and we need to do it manually. We need to add the following line to our code:

replace golang.org/x/crypto => github.com/ProtonMail/crypto v0.0.0-20190814153124-b5b07a6add54

Our go.mod file will look like this:

go.mod

module github.com/threkk/how-to-use-go-modulesgo 1.13require github.com/ProtonMail/gopenpgp v1.0.1-0.20190912180537-d398098113edreplace golang.org/x/crypto => github.com/ProtonMail/crypto v0.0.0-20190814153124-b5b07a6add54

Now we can vendor and build our project without errors:

$ go mod vendor
$ go build

Go also keeps a list of all the concrete package versions that your module uses. That list is kept in a go.sum file. This file makes sure that the builds are reproducible. You can think of it as a package-lock.json in Node.js or a Pipfile.lock in Python. We can also get a similar output using the command go list -m all which will print the list of all dependencies of the current module.

go.sum

github.com/ProtonMail/crypto v0.0.0-20190814153124-b5b07a6add54 h1:b9Mgk9zYaSxsqeaq/qCUsPBIR95BcyjzTL+uFoPBG1o=
github.com/ProtonMail/crypto v0.0.0-20190814153124-b5b07a6add54/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
github.com/ProtonMail/go-mime v0.0.0-20190521135552-09454e3dbe72 h1:hGCc4Oc2fD3I5mNnZ1VlREncVc9EXJF8dxW3sw16gWM=
github.com/ProtonMail/go-mime v0.0.0-20190521135552-09454e3dbe72/go.mod h1:NYt+V3/4rEeDuaev/zw1zCq8uqVEuPHzDPo3OZrlGJ4=
github.com/ProtonMail/gopenpgp v1.0.1-0.20190912180537-d398098113ed h1:3gib6hGF61VfRu7cqqkODyRUgES5uF/fkLQanPPJiO8=
github.com/ProtonMail/gopenpgp v1.0.1-0.20190912180537-d398098113ed/go.mod h1:NstNbZx1OIoyq+2qHAFLwDFpHbMk8L2i2Vr+LioJ3/g=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/konsorten/go-windows-terminal-sequences v1.0.1 h1:mweAR1A6xJ3oS2pRaGiHgQ4OO8tzTaLawm8vnODuwDk=
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/sirupsen/logrus v1.4.2 h1:SPIRibHv4MatM3XXNO2BJeFLZwZ2LvZgfQ5+UNI2im4=
github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.2.2 h1:bSDNvY7ZPG5RlJ8otE/7V6gMiyenm9RtJ7IUVIAoJ1w=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190422165155-953cdadca894 h1:Cz4ceDQGXuKRnVBDTS23GTn/pU5OE2C0WrNTOYK1Uuc=
golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=

To see the full example working, you can check the repository:

https://github.com/threkk/how-to-use-go-modules

Migrating existing project to Go modules

The world is not full of greenfield projects but full of maintenance. If you have an existing project, it is possible to migrate it to Go modules.

If your project does not use any type of modules

You need to run two commands in the project directory:

$ go mod init <package name>
$ go mod tidy

This will create a go.mod file and it will populate it with all the dependencies used in your project. Because your project does not have versions attached, it will use the latest version available.

If your project uses some type of dependency manager

If you were using a package manager like dep, you can rely on the modules command to import your configuration. You need to initialize the module in the same folder where the dependencies file is located.

$ go mod init <import path of your package>
$ go mod tidy  # Optional

Conclusion

Go modules are a step forward in terms of dependency management in Go, one problem that predates the language since the early beginning. From a user’s perspective, it does not offer many advantages from existing previous solutions in terms of workflow, but the biggest contribution itself is that there is finally an official tool. Love it or hate it, Go modules are here to stay and it is better to get used to it. If you want to learn more about them, I recommend to follow the official Go blog that has a really complete series of articles about Go modules.