Keep your Kubernetes save with Spacegun

Space. Looks beautiful. Has nothing to do with Kubernetes deployments.

Spacegun is a tool to manage your Kubernetes clusters, with a git-centric approach.

Everyone uses Git today. I can hardly image how one would write a piece of software without it. And I can hardly remember any alternatives to Git. This is why I feel that all parts of a project must be version controlled. Not only the application, but all the infrastructure, especially the deployment pipelines. Put all your Jenkins scripts in a Git. Make all the little knobs and valves of your AWS account reviewable. Even the manual tasks that people do on a daily basis and that should have been automated two years ago.

Tools for version controlling your platform

Yes, there are tools that do that. There is Terraform to get your infrastructure up and running. If your Jenkins is not hilariously outdated, it supports pipelines as code. You can write your code, build and deploy it with Jenkins. Or even deploy it with git hooks. And have everything version controlled and reviewable by your teammates.

But then there is this need for multiple deployment stages. From your customers point of view this is very understandable. They want a prelive environment where your product owner can accept new features. And this should better not be the live environment! You might also benefit from such an environment. You can test things out in the wild. Perhaps even on a dedicated development environment.

So we need a mechanism that knows when, how, why and to which environment software can be deployed. And while your project evolves, these rules might change. They might change from incident to incident.

This complicates things a bit!

The painful mess that was our CD pipeline

So we looked for a tool that could do that. And the tool we ended up using came with a lot of features. We knew that if we wanted to do canary releases, we would be covered. As complex as you could imagine a deployment pipeline to be, the tool could handle it. It also featured a nice web client that allowed you to drill down into your deployments and see which services are acting up again. You could fiddle with all the settings directly in the browser. This was nice. The power and the feeling of having everything under control.

Until someone, by accident, deleted the deployment settings of a critical service. And we noticed that there has not been a deployment in days. Or instead of typing in slack, someone typed into the environment variable configuration. Now our product information importer wouldn’t start anymore. Rolling back was impossible and that one guy that understands the correct settings just left the office. These are the times when you want a version controlled solution. One that allows for easy roll backs, and avoids accidents.

We investigated other solutions. Some seemed too hacky, others overengineered. But none truly had a version controlled approach. So I wrote Spacegun.

Enter Spacegun

This tool is named Spacegun because space guns are pretty awesome and ridiculous. Just look at this!

You can install Spacegun with npm install -g spacegun, to get tools for controlling all deployments from the command line. And of course a server and a client executable, so that you can have a central deployment manager in your cluster and control it from your workstation. In this article I will focus on the basic principles behind Spacegun and how working with it feels like.

Visit Spacegun Tutorial for a more technical guide

The configuration repository

The configuration of Spacegun is in YAML files and you should put them in a git repository. A running Spacegun server will periodically check the master branch for new commits and automatically pick up any configuration change.

Granted, your basic settings won’t change often or at all. Things like your docker endpoint, the available cluster and the slack hook. But the configuration repository can hold much more!

Deployment pipelines as code

You can add deployment pipelines to your configuration. A typical deployment pipeline might look like this:

cluster: k8s.live.my-app.com
cron: "0 0 0 12 * * MON-FRI"
start: "plan"
steps:
- name: "plan"
  type: "planClusterDeployment"
  cluster: "k8s.staging.my-app.com"
  onSuccess: "apply"
- name: "apply"
  type: "applyDeployment"

This YAML file describes a deployment to the live cluster k8s.live.my-app.com. The cron statement tells Spacegun to execute the pipeline at 12 am from Monday through Friday. Theplan step creates a list of updates by looking at the staging cluster k8s.staging.my-app.com. If it succeeds it proceeds to apply the plan.

Let’s assume you have some kind of incident. Feelings are hurt and policies changed. Now deployments to the live environment need the approval of the product owner and must be done manually. To do that you would simply remove the cron expression and make a pull request. Once the change is merged, Spacegun will pick it up automatically. Then any time everyone is happy with the current staging you would run your deployment pipeline with spacegun run.

Keep your cluster version controlled

Let’s say, one of your teammates wants to add a new environment variable to a service that is already deployed. She would first change the configuration files that describe the service and then make a pull request. Exactly like changes to a deployment pipeline, Spacegun will automatically apply them. The command to create such a version controlled representation of your cluster isspacegun snapshot. This command will create a bunch of YAML files in your configuration, that can be edited by your teammates.

To add a new service, you can create a YAML file that describes the deployment of the service. And after you merged it on master, Spacegun will create the deployment in Kubernetes. If you have the Slack hook configured, Spacegun will even inform you about everything it is doing in your slack channel.

So all aspects of your deployments can be managed in a single Git repository. That way, all changes to your deployments are transparent and can be reviewed by your teammates.

Where Spacegun is aiming

The goal is to automate the manual things we have to do each day. We think a good step forward is adding more steps to the pipeline. A step that probes the cluster for its health. A step that rolls back a cluster to a stable state. A step that seeks approval via slack… There are still a lot of open tasks in our backlog.

We are also experimenting with templating of configuration files. You could create one template for a microservice and one for an importer and then fill out only the parameters for the actual microservice or importer. And I am always working on stabilizing and improving the current functionality.

We started to use Spacegun in a production environment on our customers project and are happy so far. We are seeing less downtime and a better understanding in the team for our infrastructure.

And I am looking forward to hear from you at the Spacegun Github page.

Keep aiming for the stars.