Easy Team Management of Kubernetes Config

Being a two and a half person engineering team, we did not take the move to Kubernetes lightly. We did not want yet another system to manage, especially with such a small team as ours.

Our hands were tied behind our back, however: some of the software we need to run are difficult or impossible to run on Heroku, the platform we have used since our inception. There’s the consolation of having a cheaper infrastructure bill too.

We understood that the move would make our work a little more complicated. It was not until a few weeks after did we realize what those issues would be — management of cluster config by a team of developers.

After the system was setup on Google Cloud and I began teaching others how to deploy, we started to see application integrations “spontaneously” break. We soon found that we were overwriting one another’s configuration — especially secrets.

What does a programmer do when there’s a task they have to do often, not fun, and error prone? We automate!

The most painful problems for us were the following:

1. Changing secrets
2. Restarting deployments
3. Deploying a new image

Changing Secrets

Managing secrets was by far the most painful. We would have a local copy of the uncommitted secret template that we would have to copy from the Kubernetes cluster. If one of us either a) fails to update our copy and overwrites the copy on Kubernetes or b) encode it wrong (with that awful \n at the end), say hello to a new bug ticket.

kcu secret list production/worker
kcu secret get production/worker sidekiq_concurrency
kcu secret set production/worker sidekiq_concurrency 10 

Restarting Deployments

After a secret is changed, the more common action is to restart the deployments that read the secret. There isn’t a standard way of doing this, so we opted to take the advice of Luke Heidecke and annotate the deployment and push those changes. This makes Kubernetes handle then restart of the pods.

kcu deployment restart staging/website

Pushing A New Image

Finally, after the image is built, it was less than ideal to copy and paste the new image name into the config. Why not have a deploy command like Heroku?

kcu deploy master
kcu deploy staging

Ok, I’m cheating a bit. There’s a kcu.yml you need to setup in your repo.

That’s It

We considered other features like automatically watching the rollout after you deploy, but decided to pause with these three. In the interest of not over-engineering our tools, we’ll wait and see what else bothers us.

We would love some comments and especially pull requests!