Self updating tools in Go lang

In a previous post, we’ve discussed about the benefits of writing internal tools and scripts in Go , but what about handling updates? Why does it matter?

Well, we all face at some point the issue of a deployment script failing because something changed and the script wasn’t updated accordingly. It’s especially bad since it can be hard to understand on which side the issue comes from. It’s time consuming and creates distrust in the tool.

Still, it’s easy to overlook the update process, to think that just a git pull and building the new version is enough but this doesn’t scale well. If everyone needs to watch out for updates, it’s just a matter of time until someone ends up using an outdated version.

So let’s push updates toward the team rather than burdening them with mandatory polling. Our goal is to have the following process in place:

What is required to implement self updates?

In order to implement such a mechanism, a couple of questions need to be answered:

• how to package the tool?
• how to handle multiple platforms?
• where can the binaries be stored?
• how can it know it’s outdated?
• how to build new versions automatically?

Packaging and cross compilation

Using Go to write tools or scripts provides the advantage of being able to ship self contained executables (through statically linked binaries, as opposed to scripting languages for example). This takes out of the equation dependencies issues.

Still, unless everyone is working on the same OS, binaries must be compiled for each target platform. Cross compilation is never fun to deal with but there are solutions to ease things, like xgo which wraps each platform’s compiler in docker images under the hood while providing a simple CLI.

Where to host binaries?

There are multiple answers, on a server inside the company’s VPN and accessible over HTTP or sFTP, etc .. To make a decision, it’s mandatory to take in account availability, accessibility and security requirements.

In our case, AWS S3 is great choice as it provides permissions and policies we have already been using. It’s also very easy to interact with and there is an extensive library that cover its usage in Go.

How does the tool know it’s outdated?

Along the binaries on S3, a VERSION file is updated each time a new version is rolled out. So the tool downloads it on S3, compares it with its own version and proceeds to download the update if the remote version has a higher version number.

This requires internet connectivity to run the tool, but this isn’t mandatory. It’s a matter that the tool itself can decide upon, depending on its purpose.

In the case of deployments, it doesn’t make sense to deploy while being offline so this isn’t a problem in the first place, even if in our implementation there’s an option to skip the check.

Building new versions and shipping them

At Heetch, every commit on internal tools triggers a build on CircleCI (a continuous integration solution). This mechanism can be leveraged to deal with cross compiling and upload the resulting binaries on S3.

So when CircleCI performs a build, it assigns it a build number, which is just an integer that gets incremented over and over. Since those numbers are ordered as opposed to commits SHA1s for example, they can be used as version numbers.

In our case, every commits are built on CircleCI accordingly, though the only candidates for releases are the master branch builds. When it happens, the existing binaries and a file storing the current version are overwritten on S3.

Here the script we’ve been running on CircleCI that handles that part:

If we need to rollback something, we can just rebuild the tool to overwrite the faulty build while bumping the version, which is simpler than adding a rollback mechanism that could download a previous version.

Usage Example

As this process is completely generic, we’ve extracted it in a single package. It’s really simple to use as shown below.

And that’s it. Given a release had been uploaded on the specified S3 bucket, this snippet of code will replace the binary locally after downloading it.

You can find s3update that implements the solution described in this article on Github. Contributions are welcomed!