Nix and Skaffold for painless developer environments

At Immuta we have recently started to split up our Monolith Node service into micro services. As part of this process we decided to try out new technologies to make the developer experience easier and better match how we deploy our product.

Our Development pain points

One of our most frustrating issues is getting development environments spun up. While we had a README for what you needed to install, it always seemed to be out of date when a new developer would get spun up. If it wasn't out of date, our native node add-ons would sometimes not compile due to weird macOS version issues. If they could get our node service running, getting it to run in Kubernetes was an even bigger pain.

To tackle these issues were decided to adopt Nix and Skaffold. Nix to automate downloading development tools, building dependencies. Skaffold to simplify building/running our app in Kubernetes.

Nix

Disclaimer: I am far from a nix expert so I will try my best to explain the high level concepts but will mostly skip implementation details.

Nix is a package manager/build tool/functional programming language to make reproducible builds and environments. It works on both macOS and Linux. It even has a whole Linux distribution based on it. Nix sandboxes all its builds by default so you can be confident builds will continue to work in the future. The sandboxing also works “at runtime” so you can have multiple versions of a package installed on your computer at a time. You can basically think of nix as pyenv, nvm, etc for every package on your system.

The simplest way to use nix is with flakes, flakes make it easy to specify outside dependencies with inputs , and specify what the flake produces with outputs . For our new node micro service our flake looks like

Here we use the unstable nixpkg repo to make it easier to get updates for external packages. Nixpkgs has more than 80k packages already defined, in this flake we use many different kubernetes CLI programs and install node 14.

You might think its risky to use the unstable nixpkg repo but flakes has a lock file similar to package.lock, cargo.lock, etc to pin the commit of nixpkgs that was used. This way all developers will have the same versions of each package.

When a developer wants to use the flake, they can run nix develop which will activate the devShell by adding all the packages defined in the flake to your path. When you exit the devShell all the nix packages in the flake will be removed from your path so its like it was never installed (though its still on your machine in /nix/store )

The nix develop call is great but with one extra tool it can be even better.

Automatically load the devShell

Using direnv and nix-direnv, you can configure the flake to be loaded/unloaded when you cd into your project’s directory. If you run the following in your projects root directory

echo "use flake" >> .envrc
direnv allow

This will setup the .envrc (a bash script with some extra helpers) to load the flake when entering your repo. This will also re-load the flake if it detects it was changed so even when changing/pulling branches your environment will always be up to date.

Simplifying Kubernetes with Skaffold

Skaffold is a CLI tool to manage your Kubernetes cluster more easily. I think of it as docker-compose for Kubernetes. Skaffold can be configured to do file sync so if your container is setup with something like nodemon to restart your server on file change you won’t have to rebuild/redeploy your containers when developing.

My team has found running Kubernetes with docker desktop has been the simplest way to get us started. When combined with nix to install Skaffold, developers can have their full developer environment and clusters running within minutes of cloning the repo.

Setting up Skaffold

At the root of your repo you can add a skaffold.yaml that looks something like

In the requires array you can put paths to sub folders that hold a skaffold.yaml for each project/micro service you want to deploy (referred to as skaffold modules).

For example one of our folders launches a PostgreSQL container using the Bitami helm chart and it looks like this

We can add our own helm values file in the same folder that looks like this

the skaffold.yaml also references a root overrides.yaml file at the root of our repo, we do not check that file in (and create it with .envrc ) so each dev can modify their deployment without worrying about accidentally checking in their config.

Wrap up

Nix is one of my favorite new tools I’ve discovered in the last few years. While its documentation can be pretty lacking, it is incredibly rewarding when you get it working. There’s no more “you forgot to update/read the README”, everything just works. What I showed here only scratches the surface as well, you can use nix to make docker images that are incredibly slim, setup a build server to easily share builds with others, use nixos to have a fully declarative linux install, use it in CI instead of docker to speed up CI times, and much more.

When you combine nix with Skaffold you can extremely easily mimic your production environment by developing your application entirely in kubernetes. In the past deploying to a local kubernetes cluster was a real pain point for me, with Skaffold I’m always in kubernetes.

We’re Hiring!

If you also care about your developer experience, we’re hiring! If you don’t and just wanna get stuff done, we’re hiring! If you want to hear me talk your ear off about my latest programming obsession, we’re hiring!