GOLANG

Modules are a new way to manage dependencies of your Go project. Modules enable us to incorporate different versions of the same dependency without breaking the application.

Uday Hiwarale

Follow

May 4, 2019 · 24 min read

💡 Before we begin, it’s worth mentioning that Modules are supported from Go version 1.11 but it will be finalized in Go version 1.13. So, if you are using Go version below 1.13, then implementation of Go Modules might be subjected to change in the future.

Let’s talk about the pre-Go Modules era. As we discussed in the packages tutorials, in order to work with Go, we need our source code to be present inside the GOPATH directory (this is your Go workspace).

When you install any dependency packages using go get command, it saves the package files under $GOPATH/src path. A Go program cannot import a dependency unless it is present inside $GOPATH. Also, go build command creates binary executable files and package archives inside $GOPATH. Hence, GOPATH is a big deal in Go.

I always hated this as it forces us to relocate all the projects inside one directory or we have to constantly change the GOPATH environment variable to switch between different projects.

Also, you can’t install multiple versions of the same dependency package since go get puts the dependency package at the same location despite having different versions (as directory name is the same as the package name).

If you really need to work outside $GOPATH directory, then you will be switching the GOPATH environment variable back and forth. And when you do that, you need to reinstall or copy all the packages in this new directory.

Here is an interesting fact, Go does not have a central repository for downloading packages. If you are a Node.js developer, then you should be familiar with the npm command. NPM is the central registry to download and publish Node.js modules (packages).

So if Go does not have a central package registry, then how go get even works? Typically, a third party Go package import statement looks like this

import "github.com/username/packagename"

If you look at the above package import statement, the package name looks like a URL, because it is. Go can download and install packages located anywhere on the internet. To install the above package, you need to use go get command with this exact URL (without HTTP protocol).

go get github.com/username/packagename

Go visits the URL https://github.com/username/packagename and downloads the package if the URL is successfully resolved. Once the download is completed, it saves the package content inside github.com/username/packagename directory under $GOPATH/src.

As we have learned in Go installation tutorial, standard Go packages like located inside GOROOT directory (where Go is installed). Other project-level dependencies are stored inside GOPATH.

💡 If you compare that with npm, npm stores packages inside the project folder (inside node_modules directory) and not at a central location like GOPATH. With NPM, we also have package.json (along with package-lock.json) which is a text file containing packages installed for the given project and their versions.

➪ Read tutorial on Go packages from here.

Let’s understand how go get works and how to host your own custom Go repository. Like we installed a package above, when you execute go get command like below

go get domain.com/path/to/sub/directory

Go visits website domain.com/path/to/sub/directory securely using the HTTPS protocol. If the given URL does not support HTTPS or results in SSL error, and if GIT_ALLOW_PROTOCOL environment variable contains HTTP protocol, then Go attempts to resolve the package URL with HTTP protocol.

A Go package located on the internet is a version control system repository (VCS) like GIT or SVN. Supported version control systems are mentioned below.

Bazaar        .bzr
Fossil        .fossil
Git           .git
Mercurial     .hg
Subversion    .svn

If the domain.com is one of the code hosting sites recognized by Go, Go first tries to resolve domain.com/path/to/sub/directory.{type} where type can be .git, .svn etc. supported by the recognized website mentioned below.

Bitbucket              (bitbucket.org)     .git/.hg
GitHub                 (github.com)        .git
Launchpad              (launchpad.net)     .bzr
IBM DevOps Services    (hub.jazz.net)      .git

When a VCS supports multiple protocols, Go tries to resolve the URL using all the supported protocols one at a time. For example, as Git supports https:// and git+ssh:// protocols, they both are tried in turn.

If Go successfully resolves the above URL, the VCS repository is cloned (using VCS tools like Git) and saved to the relevant path in $GOPATH/src as discussed earlier.

But If the package URL is not one of the recognized code hosting sites, Go have no way to determine the VCS. In that case, Go tries to resolve the URL as it is, using HTTPS or HTTP protocols discussed earlier. If the URL resolves successfully with an HTML document, Go looks for a specific META tag mentioned below.

<meta name="go-import" content="import-prefix type repo-root">

💡 It is recommended that this meta tag should appear as early as possible in the HTML source code, to avoide any possible parsing errors.

Let’s talk about this meta tag a little.

• import-prefix: This the import path of your module. In our case it is domain.com/path/to/sub/directory
• type: This is the type of your VCS repository. It can be one of the supported types mentioned earlier. In our case, it could be a Git repository, hence git is the type.
• repo-root: This is the URL where the VCS repository is located. This should be a fully qualified VCS repository. For example, in our case, it could be https://domain.com/repos/name.git . This git the extension type is optional as we already mentioned the type as git.

Using this meta information, Go can clone the repository located at https://domain.com/repos/name.git and save inside $GOPATH/src under directory name domain.com/path/to/sub/directory

If your import-prefix is different than the URL used in the go get command, then Go will clone the repository under that directory mentioned by import-prefix value. For example, if our go get command is like below,

go get domain.com/some/sub/directory

Then, Go will visit https://domain.com/some/sub/directory and the server returns an HTML document with the below meta tag.

<meta name="go-import" content="domain.com/someother/sub/directory git https://domain.com/repos/name.git">

Since the import-prefix is different than the URL used in go get command, Go will verify if domain.com/someother/sub/directory has the same meta tag as https://domain.com/some/sub/directory/ and install the package under the directory name domain.com/someother/sub/directory and our package import statement will be like below

import "domain.com/someother/sub/directory"

💡 Usually, we should avoid this extra redirection and keep import-prefix same as the URL used in go get command.

I guess, so far we have gained a good amount of knowledge on how traditional package management works in Go. Let’s see how this can be harmful when a package becomes backward incompatible.

Let’s say that a package located at github.com/thatisuday/stringmanip currently supports string manipulation utilities like making string uppercase and all. When people download this package using go get, they are cloning the master repository at the most recent commit.

Suddenly new commits come in which either changes the implementation of the utility functions, adds breaking changes or introduces bugs. Now, when people upgrade/reinstall this package, their application will start to break.

There is no way we can force go get to point to a specific commit or release tag in the Git history (hope is not a tactic ). That means, there is no way to install a package pointed to a specific Git version.

Also, since Go saves the package under the directory name specified by the package itself, we can’t save multiple versions of the same package at the same time. This is just brutal and Go Modules are here to rescue.

Let’s first understand the mechanism of the Go modules, in theory. As we discussed some problems with Go’s traditional dependency management, there are few modifications we can easily propose.

• First of all, we should be able to work from any directory, not just GOPATH which gives us the flexibility to relocate our source code anywhere.
• We should be able to install the precise version of a dependency package to avoid breaking changes.
• We should be able to import multiple versions of the same dependency package. This is useful when our old application code is running with the old version of the dependency but we would like to use the new version of the dependency package for any new developments.
• Like package.json in NPM, we need a file that can list the dependencies of our project. This is helpful because when you distribute your project, you don’t need to distribute your dependencies with it. Go can just look at this file and download the dependencies for you.

Well, the good news is, Go modules can do all these for you and much more. Go modules gives us a built-in dependency management system. But, let’s understand what a module is first.

A module is nothing but a directory containing Go packages. These can be distribution packages or executable packages.

💡 A module can also be treated as a package that contains nested packages. You can learn about nested packages from the packages lesson.

A module is also like a package that you can share with other people. Hence, it has to be a Git or any other VCS repository which we can be hosted on a code-sharing platform like Github. Hence, Go recommends

• A Go module must be a VCS repository or a VCS repository should contain a single Go module.
• A Go module should contain one or more packages
• A package should contain one or more .go files in a single directory.

Enough theory, let’s code now. Let’s create an empty directory anywhere but inside GOPATH. I am using nummanip directory to host my module and it will contain some packages to that to manipulate number data type.

As discussed before, since a Go module should be a VCS repository, I have created a Git repository on Github with below URL.

Now, the first step is to initialize the Go Module inside this directory. For that, we have go mod init <import-path> command.

This command creates go.mod file (which is like package.json file in NPM) that contains module import path and dependencies used by packages inside it. Also, let’s initialize the Git repository with the above GitHub remote URL.

In the first step, we initialized a Git repository and pointed it to the Github repository we just created. In the second step, we initialized a Go module and specified module import path in the command.

go mod init github.com/thatisuday/nummanip

Creating modules inside GOPATH is disabled by default. You will get this error if you initiate a go module inside $GOPATH→ go: modules disabled inside GOPATH/src by GO111MODULE=auto; see ‘go help modules’.

💡 This is a preventive measure to deprecate GOPATH in the near future. If you really need to create Go modules inside GOPATH, set GO111MODULE environment variable to on. This environment variable also controls how go get should behave inside GOPATH and outside of it.

When GO111MODULE=off, the go get command simply clones the package from the master repository of the package and puts it inside $GOPATH/src. If GO111MODULE=on, then go get command puts the package code inside a separate directory and it supports versioning (discussed later in this article)

From Go 1.13, GO111MODULE is set to auto by default which means go get command inside GOPATH will act as if GO111MODULE=off and when it is invoked outside GOATH, it will act as if GO111MODULE=on.

You can use GO111MODULE=on go get <package>@<version> command to set the GO111MODULE environment variable in the command execution.

The above command created a go.mod file which contains the module import path and version of the Go this module was created on. The most difficult part is done and from here, we should focus on developing our packages.

We have created two packages inside our module. Currently, these are empty directories but soon we will put some code in there. calc package will export utilities to calculate something with numbers while transform will be used to transform numbers or number related data types (like an array of numbers).

⚠️ Since we are providing multiple packages through our module, we have created directories for each of them. But if want to just provide single package, you don’t need to create a separate directory for it. You can just write the package code inside the module directory (where go.mod file is). The only difference is how you import your package, which we will see later.

At this point, we are not sure if our module is a fully-fledged executable application or a distribution module with utility package(s). I always recommend keeping your reusable logic in separate packages and import it in your application code.

So to test our module and packages, we are going to create another module that will consume nummanip module packages. We won’t be publishing this test module as it is just for testing, hence we can init this module with a non-URL module name. We can use go mod init main to initialize the main package which will be the executable package.

💡 Go provide go test utility to test your packages with custom test suits. But that is a completely different topic which we will perhaps cover in upcoming tutorials.

We have created main module for testing purpose using the command go mod init main which created go.mod file. Also for simplicity, we have opened both main and nummanip module in VSCode as a workspace.

We have created math.go file inside calc package to provide a utility function to return the sum of two numbers.

Focus on package declaration part, package calc signifies that math.go belongs to calc package and since this package is in a separate directory under our module, package declaration has nothing to do with the module name. We have learned this inside the nested packages lesson.

💡 If our package code were inside the module directory, then package name would be same as the module name (in order to successfully resolve import statement).

Let’s publish our module. Publishing is just pushing your code to the remote VCS repository with a tag. But before we do that, we need to understand semantic versioning and git tags. So let’s get into it.

Semantic Versioning or SemVer is a universally accepted format to tag your release packages/modules etc. It has vX.Y.Z format where X is a major version number, Y is a minor version number and Z is patch version number (translated as vMajor.Minor.Patch).

For example, a package version 1.2.0 has the major version 1, minor version 2 and patch version 0. We bump only patch version when there is a small change in the package, minor in case new or enhanced functionality is added. When there is a major difference between the old version and the new version, we bump up the major version number resetting minor and patch version to 0 (for example 2.0.0).

💡Additional pre-release tags can be added as a suffix with release tag which goes like this x.y.z-rc.0 which means it is a release with release-component 0 (or x.y.z-beta.1). This is helpful to those who want to test the beta version.

Go specifies that, when there is an incompatibility between the new version and the old version, then the new version should be a major release. When a major version is released, Go treats it as a different module which we will see later in this tutorial and discuss why that is important.

As we know, a Git branch is nothing but a history of commits. Each commit has a unique identifier (commit hash) associated with them. At any specific commit, we can extract the state of the files in the repository.

When you want to release anything for public use, you need to provide a commit hash where the state of the files in the repository is stable and can be used in the production. But what we can also do is create an alias for the commit hash with the SemVer name. It’s called tagging.

Git provides two types of tags. The Lightweight Tag is simply a pointer to commit in Git History. While Annotated Tag is saved as a full object (which contains additional information like Tagger’s name, tag message, and other info) in the Git database. You can read about Annotated Tags here in depth because for simplicity, we will use lightweight tags.

Since we are releasing our module for the first time, we need to create a commit and push it to the remote repository. Then we will create a tag from the last commit (which will be the one we pushed) with a SemVer.

From above, we have created a commit and pushed it to the remote master branch. -f flag was used to forcefully push the local git history to the remote which is OK for the first-ever commit. Let’s create a Git Tag.

Since we are releasing the first version of our module, the SemVer should be v1.0.0. When you are releasing a Go module, your SemVer tag must start with a lowercase v (for successful version resolution). When you create a Git tag, you need to push it to the remote repository with git push --tags.

I have created app.go file inside main module to test our newly release Go module. Basically, we are importing the calc package from github.com/thatisuday/nummanip module to utilize Add method.

Since we know the import path of the module (and package in it), we can import directly in our code with a full URL path to the package which is github.com/thatisuday/nummanip/calc. To understand this, check out the nested packages lesson.

💡 If our package code were inside module directory itself, we would have imported it with github.com/thatisuday/nummanip only and we would have needed to use nummanip as the package variable to run nummanip.Add().

Notice that until now, we haven’t installed this module yet using go get. Yes, to install a module, you can also use go get <package-name> command or use go get command to install all packages listed inside go.mod file.

But, when we run app.go using the command go run <file>, Go fetches latest version v1.x.x (explained later) of nummanip module. When Go successfully downloads the module, it also updates the go.mod file to save the downloaded dependency.

💡 This way, we do not need to tell other people which dependencies to install. Go can just look at go.mod file for dependencies needed by the module.

⚠️ You might ask, how Go resolves the package import URL as https://github.com/thatisuday/nummanip/calc returns 404 Not Found page. I couldn’t find exact documentation on it but my guess is that since GitHub is one of the recognized code hosting sites, it knows where the package is located and this documentation suggests that.

Now, there are many questions popping in our heads. When Go will install the module automatically? What’s the use of go get then? Where are the modules stored on the system?

Go modules are stored inside $GOPATH/pkg/mod directory (module cache directory). Seems like we haven’t been able to get rid of $GOPATH at all. But Go has to cache modules somewhere on the system to prevent repeated downloading of the same modules of the same versions.

When we execute the command go run or other Go commands like test, build, Go automatically checks the third party import statements (like our module here), and clones the repository inside module cache directory.

We can see the cache directory has nummanip module tagged with version v1.0.0 to make import resolution simple. Go creates go.sum file to save checksums of the downloaded direct and indirect dependencies’ content.

Unlike package-lock.json file in npm which stores versions of the currently installed direct and indirect dependencies for 100% reproducible builds, go.sum file is not a lock file and but it should be committed along with your code in case your module is a VCS repository (explained here).

When you shared your module, go.sum plays an important role to ensure 100% reproducible builds by validating the checksum of each module.

Let’s add some patch version level functionality to our calc package.

What we’ve changed in Add function is the ability to accepts multiple arguments using a variadic function argument (read tutorial). We then pushed a new commit with tag v.1.0.1.

Let’s implement a newly improved Add function inside main module. But there is a gotcha here. Since Go already has nummanip module, go run or other commands will not fetch the newer version. To do that, we need to update our module manually (in the worst case, reinstall it).

To update Go modules present inside a module with go.mod file, use go get -u command. This command will update all the modules with the latest minor or patch version of the given major version (explained later). If you want to update only patch version even if the newer minor version is available, use go get -u=patch command instead.

If you need a precise version of a dependency module, you can use go get module@version command, for example, in our case to install v1.1.2, we should use go get github.com/thatisuday/nummanip@v1.1.2.

As you can see from the above screenshots, Go have downloaded the latest version v1.0.1 and saved in the module cache directory. This is a great way to manage dependencies on a global level, where multiple modules on the system can access different versions of the dependency.

Upgrading to a Major Version

Now the time has come to explain why Go only automatically resolves only v1.x.x versions of a dependency module or why go get -u updates only modules within the current major version.

It is generally accepted that a major version of a dependency is needed when there is a substantial change in how given dependency works. For example, Angular v1 is very different from Angular v2. They both works in very different ways.

Also, when a major version of a dependency is imported, it could lead to incompatibilities which means your old code might now work with the new major released version of the dependency.

Hence, if go get -u would have updated the dependency module to a major version for example, from v1.0.2 to v2.0.0 then our application might now have worked/built properly. Then how Go solves this issue?

Go says, when you update your module to a newer major version, technically it’s a different module as it’s not backward compatible. So module with v1.x.x is different package compared to v2.x.x. That means the consumer who has imported your module must manually install the module with the new major version using go get and specify the new import path.

So what’s the deal with a new import path? As we have go.mod file which specifies the module import path at the top, we need to change to something else. But don’t worry, the only modification we need is to append major version code (vX) to the import path so that consumer can import multiple versions of the same package. Let’s see that in action.

From the above example, we can see that, we added the functionality inside the function Add to check if arguments are greater than 2. If arguments passed are less than 2, then Add will return an error as its first return value, else sum as its second return value, which also makes sense.

This implementation of the function Add will not work with previous code as Add in version 1.x.x used to return only one value. Which means, our code has officially become backward-incompatible (not something to cheer for though). Hence, it’s fair to assume that the next release version should be v2.0.0 as it is a different package overall and Go is not wrong about that.

If you have noticed, we created a different branch v2 for the new major release. This will make thing easier to handle since we need to also support v1 in case some bugs or enhancement is needed there.

Now the most important thing is to update our go.mod file to add version prefix in the module import statement.

Go understands the vX syntax and successfully resolve the module import despite this confusing import path. vX syntax is not flexible, which means it should be precise with the released SemVer major version release number, for example, v2 for v2.x.x release.

In the above example, we have installed a newer major release version of nummanip module. Since it’s a major release, we need to use go get command to manually install it. Since this newer release also has new import syntax, we need to use that too.

The annoying things could be aliasing your package. Since we are importing two different versions of the same package, we need to alias either one of them to resolve the same package name variable conflict. Hence, we aliased v2/calc package to calcNew and used it to access Add function.

You can also see that, go get command has added module entry to go.mod file and saved the newer version of the module inside the cache directory.

Run and Build Executable Module

When you are working on an executable module, you can run the module by using go run . command or run individual files using go run path/*.go command which I have explained in my packages tutorial.

When it comes to building a module, you can either use go build or go build . command which builds the module and outputs the binary in the current directory.

You can also use go install or go install . which installs (outputs) the module binary inside $GOPATH/bin.

💡 When it comes to non-executable modules like nummanip module in our case, unlike pre-go1.11, you can’t create package archives using go install command. This is because go install can’t predict the module version (SemVer) of a module from the Git history or Git tags. Also, Go Modules do not save packages as binary archives as explained in my packages tutorial.

Indirect Dependencies

We have used the “Indirect Dependencies” term before but haven’t explained yet. Well, Indirect Dependencies as they sound aren’t direct dependencies to our module. Direct dependencies are those which our module imports while indirect dependencies are those which direct dependencies import.

go.mod files note downs both direct and indirect dependencies and mark them using a trailing comment as shown in the below example.

From the above program, we know that github.com/fatih/color is a direct dependencies since we have imported it inside our code. When we run or compile the module, Go update go.mod file and adds indirect comment to the dependencies which are not direct.

I hope this might have cleared a lot of things about Go Modules, but one question remains is that when there is no version suffix present in the module import declaration (inside go.mod file of the dependency module), which version Go fetched automatically? It’s the latest version of v1.x.x because, by default, the version suffix is v1.

Minimal Version Selection

From what we have learned so far, we can import multiple versions of the same dependency modules but as long as they are major release versions.

There is no way to import multiple modules whose versions differ by only minor or patch version (because there is no different module import statement for minor or patch releases).

Let’s say, you have a project module which has dependency modules A and B. These are completely different modules but they both have a dependency on the same module which is Module 1. But the gotcha is, Module A requires version v1.0.1 while Module B requires version v1.0.2. So each module has defined the Minimal Version of the dependency they need in order to work properly. So, if we use v1.0.1 in the final build, there is a possibility that Module B will give unexpected results or fails to build at all.

Since in the build, we can deploy only one version of this dependency, we have a Diamond Dependency Problem (as shown below).

As per Go’s recommendation, multiple versions of the same dependency with a common major version should be backward compatible. Hence, if we use v1.0.2 in our final build, Module 1 will function OK with v1.0.2 as it contains all the functionality of v1.0.1. Go calls this Minimal Version Selection (which also means to select maximal version between the dependencies). Minimal version selection is explained here in details.

So finally, we ended up with v1.0.2 of the dependency Module 1.

Go modules are in the beta stage and there might be changes in the future. Since I generally do not always get the time to follow up on these things, please drop a private note on this article if anything new comes up. This article became possible due to a private note about Go Modules only. So, any help is appreciated :)

BONUS TIPS

Local Development

When you develop a module that is supposed to be consumed as a public library, you should always write test cases to make sure every exported API is working as expected. You should follow this article to write test cases.

However, at times, your module has multiple packages such as nummanip module in our case. When a package inside the module is dependent on another package of the same module, you can simply provide the full URL of the package in the import statement such as below.

// nummanip/transform/program.go
package transformimport "fmt"
import "github.com/thatisuday/nummanip/calc"func SomeFunc() {
  fmt.Println( calc.Add(1,2) ) // 3
}

Go understands that you are trying to import the package from the same module (by looking at the go.mod file), so it won’t try to download the package github.com/thatisuday/nummanip/calc and simply refer the package from the same module source code.

In some cases, you are working on two separate modules (not the two packages from the same module) and one module depends on another module. In this situation, you would need to publish the module that is being imported.

However, Go provides replace directive for the go.mod file to reference the import path (URL) of a dependency with module an alternate import path which could also be a local file-system path.

module maingo 1.13require github.com/thatisuday/nummanip v1.0.1replace github.com/thatisuday/nummanip => /path/to/local/nummanip

In the above go.mod file, the github.com/thatisuday/nummanip module will be referred from the local system path /path/to/local/nummanip. So when the module is being built, Go would use the nummanip module source code from the /path/to/local/nummanip location instead of downloading the module github.com/thatisuday/nummanip.

We can also provide an alternative remote path or a specific version of the dependency with the replace directive when the module is being resolved (while downloading). This process of resolution is documented here in detail.

You can either edit the go.mod file manually to insert the replace statement or use the go mod edit -replace command as shown below.

go mod edit -replace "github.com/thatisuday/nummanip=/path/to/local/nummanip"

Go also provides -dropreplace flag with the go mod edit command to remove a replace statement from the go.mod file. To learn more about the go mod edit command, use the go help mod edit command.

Make sure to replace the replace directive from your go.mod file before publishing the modules unless required.

Tidy Module

If you have a module that you wish to publish but its go.mod file does not track the dependencies that might have been imported inside the source code then you should run go mod tidy command.

This command adds dependencies to the go.mod file if they are imported inside the source code and remove dependencies from go.mod if they are not used in the source code. So ideally, you should always run this command before publishing the module.

The ./... pattern matches all the packages in a module. So when you run go build ./... or go test ./..., you are building or testing all the packages inside a module (including module itself if it is a package). These commands also download and install dependencies inside go.mod file.

The go mod graph shows you the dependencies of your module.

Vendoring Dependencies

Sometimes, when you are running automated tests for your project (executable module like main in our case), there is a chance that your test machine may face some network issues while downloading the dependencies or when a test machine is running in an isolated environment.

In such cases, you need to provide dependencies beforehand. This is called vendoring. You can use the command go mod vendor to output all the dependencies inside vendor directory (inside module directory).

When you use go build, it looks for the dependencies inside module cache directory but you can force Go to use dependencies from vendor directory of the module using the command go build -mod vendor.

Installing CLI modules

To install a CLI application (module) from anywhere in the terminal, use the following command.

$ GO111MODULE=on go install <package>@<version>

This will install a module inside the module cache directory and generate a binary executable file inside $GOPATH/bin directory.

Even though you could install multiple versions of a module, you can only have a single binary executable file. So the module cache directory will have source code of multiple versions, however, the bin directory will have only one instance of the module binary.

If you want to register your package on pkg.go.dev, please follow these instructions. Go uses comments-based syntax to generate documentation which also appears on pkg.go.dev. Follow these instructions to write documentation for a Go module. For reference, you can follow the Commando module I wrote to generate CLI applications.

Watch this amazing video at GopherCon 2018 to understand Go Dependency Management with versioning in simple language.

The official Go documentation on Modules is available here.