TLDR
A guide on how to set up your own S3 nix binary cache using Minio S3 server.
Introduction
What is nix?
Nix is a package manager for Linux and other Unix systems, aiming to manage packages reliably and reproducibly. To do so, it introduces Nix expressions, a functional language to build the packages. To increase reproducibility, a package description includes every step to build a package from source. This means that not only the wanted software, but also all its dependencies, are built from scratch. Therefore, a complete dependency chain can be ensured, which makes the package build process truly reproducible.
Additionally, multiple versions of the same software package can be installed on a system, making it possible to satisfy various dependency chains at a time. This enables the installation of multiple packages that depend on the same third package, but on different versions of it. To achieve this, all packages are installed in the global nix store under “/nix/store/” and are then symlinked to the respective locations. To verify a package’s uniqueness, its whole directory is hashed, and the hash put into the name of the package’s main folder. Every software built from the same Nix expression that uses the same dependency software versions results in the same hash, no matter what system it was built on. If any of the dependency software versions were changed, this will result in a new hash for the final package.
Using symlinks to “install” a package and link all the right dependencies to it also enables atomic updating. To make this clearer, let’s think of an example where software X is installed in an older version and should be updated. Software X is installed in its very own directory in the global nix store and symlinked to the right directory, let’s say “/usr/local/bin/”. When the update is triggered, the new version of X is installed into the global nix store without interfering with its older version. Once the installation with all its dependencies in the nix store is completed, the final step is to change the symlink to “/usr/local/bin/”. Since creating a new symlink that overwrites the old one is an atomic operation in Unix, it is impossible for this operation to fail and leave the package in a corrupted state. The only possible problem would be that it fails before or after the symlink creation. Either way, the result would be that we either have the old version of X or the newly installed version, but nothing in between.
Why should I use it?
Nix is great if you have a need for reliable and reproducible package installation, such as in dev or build environments. Further, it is great for testing new versions of a software, since having multiple versions at the same time is no big deal.
And what is wrong with it?
No matter how great every aspect of Nix sounds, its design has also a major drawback, which is that every package build triggers the build process for the whole dependency chain from scratch. This can take quite a while, since even compilers such as gcc or ghc must be built in advance. Such build processes can eat up a remarkable amount of memory, introducing an additional hurdle if one wants to use it on restricted platforms such as the Raspberry Pi.
Nix binary caches
To address this major disadvantage, nix package binaries can be cached. This way, the installation process is much faster since the desired software and its dependencies only have to be downloaded instead of built. In fact, a lot of commonly used software is already populated in the public caches of the nix projects on GitHub (https://github.com/NixOS/nixpkgs). Unfortunately, there is no guarantee that a package that was once available in the public caches will stay there forever. Storing so many versions of different packages consumes quite a lot of storage, and therefore older packages can be purged in favor of never versions of it.
Build your binary cache
To overcome the drawback of always building everything from scratch and the chance of losing access to prebuilt versions of packages, it is also possible to build your Nix binary cache using an S3 server such as Minio (https://min.io/).
The process of setting up your cache, populating it with binaries and use the binaries from your newly built cache will be described step by step. For this manual, I assume that you already have nix in place, so I will not cover the nix installation process. For further information on how to install nix see https://nixos.org/download.html.
Unfortunately, I did not manage to build a private binary cache, that is only accessible with the right credentials. If anyone got this working, I would be happy to add the necessary steps to this manual.
Setting up and configuring S3 server
In this manual, we make use of Minio S3 server, which can be deployed as a single instance Docker container up to a distributed large scale multi container solution. I make use of docker-compose to set up a single instance S3 server. There are further steps necessary to make the server available on the internet, which I will not describe in this manual.
docker-compose.yml
To manage the Minio server we use Minio Client which can be installed using the following commands:
macOS:
Linux:
The configuration file for the Minio Client is located under “~/.mc/config.json” and can look like this:
Then we create a bucket, a dedicated build user which can upload content to the S3 server and enable anonymous downloads.
The nix-cache_write.json should look like this:
Then set the permission to the user and enable anonymous downloads:
Configure build environment and build to S3 nix cache
On the machine we use to build the nix packages, we need the S3 credentials in places. Set up a file “~/.aws/credentials” and populate it with the credentials of our nixbuilder user:
Further, we need the nix configuration under “~/.config/nix/nix.conf” that should look as follows:
Be aware that we just disabled signed packages for our cache. If we want to make use of signed packages, additional configuration is needed.
For build demonstration purposes, we will build a chromium package with a certain version (83.0.4103.106) that I got from the public nix build server. In our case, the nix file (chromium.nix) looks like this:
To build and upload the binaries to our newly created cache, simply use the following command:
Install binaries from your newly created S3 nix cache
We want to use the binaries from the S3 nix cache on the host, so we also need nix installed and the following configuration file under “~/.config/nix/nix.conf”
Having this configuration, we always use our nix binary cache as well as the public nix cache to query for software we want to install.
Additionally, we need the same nix file (chromium.nix) we used to build and upload our chromium nix package:
To install the binaries from either the public cache or our cache if it does not exist in the public one, we call:
Summary
We saw how we can set up a Minio S3 server and configure it for the usage as a nix binary cache. In a second step, we saw how we must configure the build machine to build directly into our S3 binary cache server. And finally, we configured our client machine on which we downloaded the binaries from our S3 nix cache to install without any build activity.
Alternatives
This manual only covers a nix binary cache that is available without any credentials. The steps to achieve this are not very complicated, but it needs some background know-how and infrastructure in place. For anyone who wants to go straight forward without taking care of building his own S3 nix cache server and probably even having the need for a private and secured nix cache, I suggest taking a look at cachix (https://cachix.org/). Cachix provides the functionality to build and push nix packages to its cache and download them where needed in either freely available public caches or a paid private cache.
About the author
Remo Höppli is Co-Founder and Software Engineer at Earlybyte.
Earlybyte is an IT consultancy firm specialized in developing new digital solutions for companies around the world from digitalization to IoT solutions, close to the client and its business embracing agility.
