Go 1.5's vendor/ experiment

freeformz
freeformz
Jul 9, 2015 · 4 min read

Go currently (pre 1.5) doesn’t provide any built in method for vendoring packages. The tools that currently exist today (godep, nut and a few others) basically exploit the implementation details of $GOPATH. Go 1.5 however includes a “vendor experiment”. With this experiment `go` commands will attempt to resolve dependencies in `vendor/` directories.

Russ Cox explains it fairly well in his commit message:

If there is a source directory d/vendor, then, 	when compiling a source file within the subtree rooted at d, 	import "p" is interpreted as import "d/vendor/p" if that exists.

When there are multiple possible resolutions, the most specific (longest) path wins.

The short form must always be used: no import path can contain “/vendor/” explicitly.

Import comments are ignored in vendored packages.

To utilize this feature you need to set the environment variable `GO15VENDOREXPERIMENT` to `1` before running any of the go commands (run, test, build, install, etc)

For instance given a source tree like this:

./main.go
./vendor
./vendor/a
./vendor/a/a.go
./vendor/b
./vendor/b/b.go
./vendor/c
./vendor/c/c.go
./vendor/c/vendor
./vendor/c/vendor/a
./vendor/c/vendor/a/a.go

In this example I have main.go requiring all of the packages like so:

package mainimport (
“a”
“b”
“c”
“fmt”
)
func main() {
fmt.Println(“main.main()”)
fmt.Println(“main: a.A ==“, a.A)
fmt.Println("main: a.Inc() ==", a.Inc())
fmt.Println(“main: b.B ==“, b.B)
fmt.Println(“main: c.C ==“, c.C)
}

vendor/a/a.go looks like:

package aimport “fmt”var A = 1func Inc() int {
A++
return A
}
func init() {
fmt.Println(“Init package a”)
fmt.Println(“a: A ==“, A)
}

vendor/b/b.go looks like:

package bimport “a”
import “fmt”
var B = 2func init() {
fmt.Println(“Init package b”)
fmt.Println(“b: a.A ==“, a.A)
fmt.Println(“b: a.Inc() ==“, a.Inc())
}

vendor/c/c.go looks like:

package cimport “a”
import “fmt”
var C = 3func init() {
fmt.Println(“Init package c”)
fmt.Println(“c: C ==“, C)
fmt.Println(“c: a.A ==“, a.A)
fmt.Println(“c: a.Inc() ==“, a.Inc())
}

and lastly vendor/c/vendor/a/a.go looks like:

package aimport “fmt”var A = 100func Inc() int {
A++
return A
}
func init() {
fmt.Println(“Init c/vendor/a”)
fmt.Println(“c/vendor/a: A ==“, A)
}

Running main.go without the vendor experiment produces what you would expect:

$ go run main.go
main.go:4:2: cannot find package “a” in any of:
/usr/local/go/src/a (from $GOROOT)
/Users/emuller/go/src/a (from $GOPATH)
main.go:5:2: cannot find package “b” in any of:
/usr/local/go/src/b (from $GOROOT)
/Users/emuller/go/src/b (from $GOPATH)
main.go:6:2: cannot find package “c” in any of:
/usr/local/go/src/c (from $GOROOT)
/Users/emuller/go/src/c (from $GOPATH)

But running main.go with the vendor experiment turned on produces:

$ GO15VENDOREXPERIMENT=1 go run main.go
Init package a
a: A == 1
Init package b
b: a.A == 1
b: a.Inc() == 2
Init c/vendor/a
c/vendor/a: A == 100
Init package c
c: C == 3
c: a.A == 100
c: a.Inc() == 101
main.main()
main: a.A == 2
main: a.Inc() == 3
main: b.B == 2
main: c.C == 3

Let’s unpack that a little….

main.go imports all three top level dependencies: a, b & c.

Each of those dependencies uses an init function for demonstration purposes. As each init function is executed it prints out its exported variable. When it imports package a it also calls a’s exported Inc function so we can observe a side effect.

First we see package a being initialized:

Init package a
a: A == 1

Then we see package b being initialized:

Init package b
b: a.A == 1
b: a.Inc() == 2

Package b imports package a and because of the dependency resolution policy of longest path wins (see Russ’ comment above) vendor/a is what is used. a.Inc is called, which increments a’s internal variable.

Then we see package c being initialized:

Init c/vendor/a
c/vendor/a: A == 100
Init package c
c: C == 3
c: a.A == 100
c: a.Inc() == 101

Package c also imports a package a, but unlike package b it has its own version of a package a inside of its own vendor directory making the full path to the package a that package c imports being: vendor/c/vendor/a. This new package a has a different value for its internal variable (100) and calling a.Inc updates it to 101.

Lastly main’s main function runs:

main.main()
main: a.A == 2
main: a.Inc() == 3
main: b.B == 2
main: c.C == 3

Showing that the package a used by main is the one in vendor/a, not the one in vendor/c/vendor/a or vice versa.

Recursion!

Side Effects + Gotchas

Tools like godep, vendor, gb, etc may be called upon to “squash” down recursively vendored packages into a single version in the vendor/ directory. It’s not really clear what people’s expectations here are though.

There is a strong argument to be made that libraries should not vendor their own dependencies, but as the author of several library / command combo packages that rely extensively on 3rd party libraries I’m not convinced of that blanket assertion.

Determining the proper versions without a bunch of extra metadata around will be hard. This is probably a place where the vendor-spec comes in handy.

In any case I am looking forward to the official support for this.


Originally published at icanhazdowntime.org.

Thanks to Ines Sombra.

freeformz

Written by

freeformz

I am too many things, yet not enough.