Managing proprietary vendor dependencies in Golang

SoftWare Samuraj
Jul 20, 2018 · 5 min read

New challenges arrived after my career change — being a long time Java developer based in “standard” enterprise environment, I switched recently to the area of cloud and big data development.

We are now prototyping a new product and Golang has been chosen as a primary technology. I think it’s a good choice because of nature of the product — it’s a bunch of microservices build on top of IaaS.

Golang is not so new language — it’s here around 9 years — and although it’s pleasant to write in it, it bears some pitfalls and challenges if you want to use it in a little bit different way than it was initially designed (or if you expect some features common in more mature platforms).

Me and my team, we all have strong Java background and therefore it’s possible that our approach to Golang infrastructure is not adequate. Nevertheless, I spent last three months with intense learning and research of topics like versioning, dependencies, reproducibility and automation of the Go builds and I would say that I found some gaps.

Requirements

We had couple of rather common requirements:

  1. Proprietary Git repository behind a company proxy. Because of legal issues and company policies, we can’t have source code on GitHub.
  2. Standalone project repository. OK, I got it —in Google, they love monorepo. But majority of us don’t work in Google.
  3. Reproducible build. No-brainer.
  4. Reliable dependency and version management. Common thing on Linux or in Java — all you need to have is a repository and some package manager, right?
  5. Fully automated. Again, no-brainer. But I still encounter people who forge (even production) builds manually.

Problems

You will encounter a problem if you want to implement above mentioned requirements in Golang: out-of-the-box, you can’t. The essential difficulty is that the whole Golang ecosystem is designed like a huge open-source monorepo (a.k.a. GitHub). If you don’t want/can’t have your code on GitHub, you have a problem. Because:

  • Golang doesn’t have a uniform/dominant tool for automation. The majority of GitHub projects builds via make, which is a fine tool for compilation. But using make is a masochism if you want to have more complex automation.
  • Golang hasn’t (yet) solved dependency versioning. One would say that after all those years it would be somehow settled. But you can take a look into the official wiki page, lapidary called Package Management Tools, to find out that there are about 20–30 such tools. Fortunately, the dep tool catches up massively from the last summer and very promising is vgo — an official proposal now, which should be part of the upcoming Go 1.11 (maybe this autumn?).
  • Official Golang tools can’t work with dependencies which are not in public repositories (like GitHub, Bitbucket, GitLab). The only supported way is to set the go-import meta tag on your repo, but very often this is not under your control.

Solution

The truth is that the solution, we iterated so far, was build with use of tracing bullets — at the same time to solve business requirements (why are we doing the prototype/product) and build automation.

For automation, we chose Gradle. Formerly, we count with some polyglot solution (partly in Java, partly in Golang), but eventually, we end up with pure Golang modules.

It’s ironic, that there is no Java in the project now (the only exception are Gradle plugins). Nevertheless, Gradle proved itself well — we build Go binaries, RPM packages, Docker images, virtual machine images, downloading and re-packaging binaries from internet, run integration tests, deploy to repositories and cloud; all with help of Gradle.

Photo by Irina on Unsplash

That’s all nice, but how can we compile Golang source code? Of course, we take some already existing Gradle plugin. Unfortunately, above mentioned problems with versioning of Golang dependencies caused that none of those plugins has been fully suitable for us.

And therefore I wrote my own plugin. I told myself that basic Go tools work well and I need just to orchestrate them. So, my plugin is nothing more than just wrapper + orchestration + lifecycle. The plugin addresses following things:

  1. GOPATH manipulation, so the tools which expect source code in GOPATH are not confused and don’t protest.
  2. Download of external dependencies from public repositories via dep tool.
  3. Download of external dependencies from proprietary repositories via Git cloning.
  4. Tests launching.
  5. Binary compilation.

The dep tool cares about version management of public dependencies. The plugin takes care of version management of proprietary dependencies (the task proprietaryVendors). In case that proprietary dependencies have transitive dependencies on public dependencies, it’s combination of both approaches.

The technical explanation is a little bit complex, therefore I refer to documentation: How to handle proprietary vendors, but the usage should be quite simple.

Sequence diagram of plugin build lifecycle

So, problem solved?

The new Gradle plugin meets all aforementioned requirements and after those past three-four months, it scored hundreds of successful builds (a build in GitLab Pipelines is triggered for every git push). So yes — it works to our satisfaction.

The question is, how long does it last —dependency versioning is hot topic in Golang community and a lot of people looks forward to slowly approaching vgo. Therefore, everything could be completely different in half a year and my plugin won’t be needed anymore.

But I’m fine with that — I already wrote few such plugins: they solved my needs in given moment, but they became obsolete later. Actually, I like it — tools which are flexible enough and have nice and modern design at the same time. And it’s fun to use them. I believe that like Golang, so Gradle, they both meet such criteria.

Links

Mind Map

 by the author.

SoftWare Samuraj

Written by

SoftWare Samurai & (soon-to-be) Golang ninja. Avid reader & marathon runner.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade