vendor directory in Go

How the hell can we use specific locked versions of libraries, without breaking the entire app? Use $GOPATH they said. 🤔

Hello and welcome everyone, my name is Steve and I am the creator of GopherTuts YouTube channel. Before reading this article, consider checking it out, if you want to stay up to date with what I do.
There’s also the video tutorial version of this article, so be sure to check out vendor directory in Go tutorial.

The Problem

So, when Go programming language landed on the stage of programming languages, apparently the language did not have anything which would allow versioned package management as we all know it today.

Well, Go always had the go get tool which downloads third party software. However, with the initial release of this tool, it used to have a drawback for a very long time, up till recently. It only used to download the latest version of the code, the code from master branch.

Now, while there are a lot of projects which are pretty stable and do not introduce breaking changes, there are other projects (third party) code which do introduce breaking changes, meaning we have to make sure our applications use a specific locked version of that library, otherwise things get pretty unstable very fast.

As of Go 1.5 the Go team came up with an approach called the vendor directory. This approach allowed the community to develop different package managers such as depor glide which would solve the problem.

So before we talk about the vendor directory, let’s make some space and quickly talk about the problem.

So the problem is versioning. How do we get specific locked versions of open source software, without ever breaking our applications.

Well, we all know the answer right? Use SemVer. But, what the hell is semver?

SemVer

So let’s assume in our project we need lib1 and lib2 imported. So we downloaded and imported the libs and our application works perfectly fine. However, all of the sudden after couple of months, our application even fails to compile successfully.

Now why is that? Well it’s simple. When we downloaded both lib1 and lib2 we downloaded the code from master branch.

After couple of months the master branch already contains breaking changes, and we still download the code from master, whenever we built the application, which is the reason why this happens.

That means all we have to do it somehow lock the version to the one which is compatible with our application, till we can successfully upgrade.

WAIT a minute. But how are we supposed to know if a library has breaking changes or not? 🤔 Well that’s exactly the problem SemVer (Semantic Versioning) solves.

So the idea is very simple, every library in its life cycle has:

• breaking changes
• features or improvements to the current feature set
• bug fixes or patches which solve certain issues about the library

When a new version of the library is released, basically the version has to be bumped accordingly to let developers and automation tools know, whether an update can be performed automatically or further refactoring is needed.

So the let’s assume we have the version: v3.1.2 of a specific library. The idea is very simple and it’s absolutely just a convention.

The version of a library or project is divided in 3 subversions:

Major

The first number: 3 v3.1.2 from the version string, indicates the major version. If this number of the library is bumped, it means the library now has a breaking change from previous major version 2. Such upgrades need to be made very carefully to maintain compatibility with the application.

Minor

The second number: 1 v3.1.2 from the version string, represents the minor version, which means, the library or project does not have any breaking changes and it adds some new functionality to it, so it’s supposed to be safe to upgrade even automatically.

Patch

The last number: 2 v3.1.2 from the version string, represents the patch or some bug fixes which were added to the library or project. Normally this is also not a breaking change so the developers or the automation tool should be able to upgrade the version without breaking the application.

Go 1.5 vendor experiment

So now that we all know the problem and we know how to solve it using SemVer(Semantic Versioning) I think it’s time to talk about the vendor directory experiment.

Let’s now imagine you have the following piece of Go code:

package main
import (
    _ "github.com/gophertuts/go-basics/vendor-directory/pkg1"
    _ "github.com/gophertuts/go-basics/vendor-directory/pkg2"
    _ "github.com/gophertuts/go-basics/vendor-directory/pkg3"
)
...

And you have the following directory structure:

So according to this directory structure, it seems unclear which package from which location is going to get used. Are we going to use

$GOPATH/package3 or vendor/package3 ??? 🤷‍♂️

Well according to how the vendor directory works, the code from vendor/package3 will get used.

Note: The vendor directory structure is the same as $GOPATH/src directory structure.

Example

So basically the Go tool will look recursively down, and the code located in the most nested vendor will get used. Here’s a little diagram to make things clear:

So basically if there’s no vendor directory present anywhere, the code from $GOPATH will get used. Otherwise the code from the closest vendor directory will be used. The Go tool will scan the $GOPATH recursively up, relative to the file that imports the package.

In that case, can we have a bunch of nested vendor directories? Technically we can, but it’s totally a bad practice and nobody really uses vendor for that purpose.

Note: when having multiple versions of the same package it is said that the package imported from vendor directory shadows the package from $GOPATH Also in order for import shadowing to take place, make sure to have the same package reference when importing the package.Otherwise you’re just importing a different package.

Import Path Checking

Besides having a long import path which usually looks something like:

import "github.com/user/project" 

We could have a path which looks like this:

import "gophertuts.com/lib"

Now, of course a lot of companies or users who develop libraries or even personal projects would love to have a custom import path for their libraries or external projects. And also having a custom import path benefits us with:

• not being tied to a host provider, today it could be GitHub, tomorrow it could be GitLab or BitBucket, and it does not even have to use Git and the import path stays the same, absolutely no change or breaking clients.
• To a certain degree it looks better and types & reads easier

However there is a problem with that. Now that you have a custom import path, clients can have multiple import paths in their projects. They can have the path like the following:

import "github.com/user/project"

or they can choose to import it like this:

import "gophertuts.com/lib"

So how do you prevent client from importing your library in multiple ways, without future breaks?

Well, there’s such a thing called import path checking. It pretty much looks like this:

So basically inside your library file you add a comment right after the package declaration saying this is the way all the clients will import this library, otherwise their code will not even compile.

And when developers will import that package the Go tool will go ahead and download the source code from the current host or provider then run it through the import path checker.

So if the cloud provider(host) changes the import path does not change. So basically the import path will always looks like this:

import "go.uber.org/zap"

No matter what the import path does not change, the maintainer may decide to move the code to GitHub or GitLab or BitBucket or his own Git server and the import path will still remain unchanged. 😎

import path checking is disabled for vendor directory

While this is an amazing feature, it’s DISABLED for vendor directory. So any source code outside vendor directory is checked against import path checking when compiling it (your source code).

If import path checker fails, the Go tool will not generate a binary, even if you import the library from the real host. It will only work when importing the official path (the one indicated in the package declaration)& it’s DISABLED for vendor directory.

go get does not update the vendor directory

When running go get the Go tool only makes sure to update $GOPATH Updating the vendor directory is the responsibility of a package manager which knows how to work with that, so go get will not touch vendor at all.

Module aware feature (Go modules)

I’ve mentioned earlier that when using go get , the tool downloads the latest master version of the code, which is true.

However starting with Go 1.11 we have Go modules, which means Go natively supports modules and semantic versioning, without needing to use the vendor directory at all.

All you have to do is have at least Go ≥1.11 and set GO111MODULE=on which will enable the module aware feature of the Go tool, giving you the possibility to install specific locked versions of libraries without breaking the applications or needing vendor directory to solve that problem.

With the help of Go modules, we can even develop Go projects outside $GOPATH , which makes developing even more flexible.

The vendor directory still holds its purpose and there are a lot of projects out there, if not most of them, which still makes it relevant.

This is not a tutorial about Go modules or module aware system in Go, so make sure to stay tuned for a future tutorial on Go modules.

Resources

This tutorial/article is only about the vendor directory, however here are a list of resources which I slightly talked about and did not get into details. Enjoy:

• Go 1.5vendor experiment
• Vendor directories
• Import path checking
• Go project structure
• Go project layout
• Go modules
• VGO Module
• GopherTuts GitHub

Conclusion

If you found this article useful, or you liked the YouTube tutorial, make sure to like/dislike, clap and share this article or the video so that others may use it. It really means a lot to me, and the feedback makes me grow and do better.

Speaking of feedback, if you have any suggestions on how to improve GopherTuts or simply wanna stay in touch with me, make sure to leave it at feedback.gophertuts.com

Make sure to also check out the GitHub repo for the source code and presentation slides used in this article & the YouTube tutorial.

Thank you for your time and subscribe to never miss a video or article about Go. Stay tuned! 🚀🚀🚀

Keep in touch

• Twitter
• Facebook
• Instagram
• GopherTuts Discord
• YouTube Tutorial