Go as a scripting language

At Heetch, in order to handle growth, we went from a Rails application to a bunch of Go services running in containers, so we had to rewrite scripts that handled deployments.

First iteration consisted of basic Bash scripts and it did the job perfectly. Without surprise, with each added feature, code inevitably became more complex, making it harder and harder to maintain.

This is usually around this point that it sounds really good to rewrite those with a dynamic language, such as Python or Ruby as a second iteration. But at that time since we started writing a lot of Go for our new services, what about writing those scripts in Go instead?

Is Go fit for scripting?

Go is clearly more verbose than many other popular languages. In that regard, it looks much closer to C than Ruby does but its simplicity, being at the centre of its design, makes it really smooth to code with.

In a lot of ways it reminds of Python and its famous “There is only one way to do it” but changes the game by being statically typed and offering great tooling to compensate the lack of flexibility.

Proper integrations and a vibrant ecosystem

First requirement is to have a proper environment to develop with. Every code editor now have proper Go integration, such as code completion, jumping to definition, renaming functions across files, etc … Those helps tremendously in being productive.

Next, critical requirements for scripting, is there enough packages out there that will allow to focus on the plumbing rather than the details?

Go has matured a lot and grew a vibrant ecosystem in which there’s a fairly high chance that a package for a DevOps related feature exists.

Just to name a few, AWS, Slack and Cobra (proper option parsing and more) are well maintained and are rather easy to use.

How about uploading a file on S3 ?

Nothing fancy, but pretty clear. One could say that its shell counter-part aws s3 cp is even easier but is the aws binary available on the developer’s machine running the script?

With Go, we can simply ship a statically built binary and voilà. No dependencies, nothing but a wget on the target machine and it can upload to S3.

Case Study, a deployment script

Now we’ve seen why it can be a good idea to write scripts in Go, let’s have a look at an example based on our deployments scripts.

How are our services provisioned is defined by a service manifest which in essence can be reduced to:

{
  "id": "my-service"
  "docker-image": "my-service:{{SHA1}}",
  "cpu": 0.5,
  "memory": 256
}

When a service is being built, its Docker image is tagged with the latest commit’s sha1. And to deploy them, it’s simply a matter of posting this JSON on http://SERVER/deploy/:service_id, while replacing {{SHA1}} with the one corresponding to the specific commit that needs to be deployed.

The containers orchestrator then finds a server in our cluster that has enough resources available to launch this service, pulls the Docker image and starts it there.

Writing a shell script for such a task is a matter of mixing jq to parse JSON, with sed and curl making this a rather straightforward script to write.

What could go wrong?

What if jq isn’t installed on the developer’s machine? Are we checking that the manifest is valid? How would we handle anything other than a 200 OK from the containers orchestrator?

The script would grow and would start to be hard to read and to maintain, especially if going down the road of deploying things in parallel in the future.

So instead of looking at Python or Ruby, let’s have a look at what Go can offer there.

Creating a nice API

There’s no point in doing all of this in Go if that’s not to provide an clean set of functions that abstracts deployments.

Basically it’s about writing func Deploy(serviceName, sha1) error then having a way to find the right manifest for that service name with func FindManifest(serviceName string) (*Manifest, error)and finally func RequestDeployment(m *Manifest) error to effectively deploy it through a HTTP request.

The first iteration in Go would then look like this:

Nothing really fancy is going on here, it’s merely about writing a couple of functions based on what we would have written in Shell script. A bit of plumbing is missing for the sake of brevity.

There’s also no specific packages structure nor any interface, the idea is to start small and see how things will evolve. Trying to guess that is usually a recipe for premature design and leads to an overly complex code which would defeat the whole purpose of simplifying the shell scripts.

Requirements such as handling deployment failures, Docker images presence checks, etc … will flow in and it will now be much simpler to introduce such changes.

So what’s next?

So far we’ve been pretty happy with writing our scripts in Go, we have since then enhanced a lot this script: services manifests are now living directly on Github rather than a local copy on the developer’s machine, notifications are posted on Slack, etc …

Someone who isn’t familiar with Go could find that writing such scripts a bit verbose but the resulting predictability and maintainability are clearly worth the shot compared to shell scripts.

And compared to Ruby and Python, benefits are there too, how Go forces the developer to structure things feels like an easier path to build and maintains such scripts. Things like mandatory error handling or strong typing are helping a lot to turn these scripts into maintainable code.

We’ve since then brought all our scripts into a single binary which then reminds a lot of Heroku’s CLI tool to interact with its services. It’s been a pleasant experience to write those with Go, its idioms guiding how to structure it along the way.

The binary is stored on S3 so all our scripts are just a wget away form the developers without any other form of requirement.

And as a side-effect, making this binary dealing with its updates is now a low hanging fruit given how simple it is to interact with S3 in Go.