Dapp tools and the Nix package manager

Why we distribute our developer tools with Nix; why it’s not just another package manager; and why you should try it if you haven’t already.

Part of the Nix definition of our dapp tools

Here’s how Eelco Dolstra motivates Nix in his 2006 Ph.D. thesis, The Purely Functional Deployment Model:

The development of principles and tools to support the deployment process has largely been relegated to industry, system administrators, and Unix hackers. This has resulted in a large number of often ad hoc tools that typically automate manual practices but do not address fundamental issues in a systematic and disciplined way.

Nowadays the popular tools for reliable distribution and deployment are things like npm, apt, Docker, and Ansible. But these don’t really solve the problem of defining how to build and combine software artifacts.

Dolstra continues:

[The problem] is evidenced by the huge number of mailing list and forum postings about deployment failures, ranging from applications not working due to missing dependencies, to subtle malfunctions caused by incompatible components. Deployment problems also seem curiously resistant to automation: the same concrete problems appear time and again. Deployment is especially difficult in heavily component-based systems — such as Unix-based open source software — because the effort of dealing with the dependencies can increase super-linearly with each additional dependency.

This thesis describes a system for software deployment called Nix that addresses many of the problems that plague existing deployment systems.

We think that Nix does in fact advance the state of the art, and we’re happy to be using it for our dapp development tools.

Let’s see how to use it in practice as an end user before going into how it works.

Basic setup

Installing Nix and our tools on your GNU/Linux or OS X system is unobtrusive and easy to completely undo should you so wish. First run the Nix installer:

$ curl https://nixos.org/nix/install | sh

The installer will tell you to add a line to your shell configuration. Do that, then continue:

$ nix-channel --add https://nix.dapphub.com/pkgs/dapphub
$ nix-channel --update
$ nix-env -iA dapphub.dapp
$ dapp --version
dapp 0.5.13

Because of Nix’s design, the dependencies of dapp are kept encapsulated. Exactly one command will be exposed in your PATH without any other visible impact on your system outside of the /nix store.

To make the whole dapp tool suite available, including the latest solc, do:

$ nix-env -iA dapphub.{dapp,seth,hevm,solc}

Our channel has signed binary packages for Linux and OS X and features

• the latest versions of our developer tools,
• versions of the Solidity compiler going back to 0.4.5,
• the latest Geth, and
• a growing set of Ethereum-related tools.

If you have any problems or questions, join our chat and we’ll be happy to help. There’s also the Nix manual with details on using and developing with Nix.

How Nix is different

All files installed via Nix are guaranteed to end up under a single self-contained directory, usually /nix. Files in the /nix store are treated as immutable.

If you look in the Nix store after you've installed some packages, you'll see that the paths include hashes. These hashes are derived from the expansion of each package including its build script and all its transitive dependencies.

This reliance on hashes is the crucial insight that makes Nix so robust and flexible. As enthusiasts of decentralized technologies, we recognize this leap from Git, and of course from blockchains.

Content addressing is now a fundamental principle of engineering, and Nix is how that plays out in package management.

There’s no problem with having several different versions of a library, or variants with different compile-time flags. Each variant gets its hash and each program links precisely to the ones it needs. And if several packages depend on the same variant, there’s no duplication in the store.

The Nix language

Nix is actually an implementation of a package specification language. That language happens to be “purely functional.” A package is defined as a function that takes its dependencies as arguments and returns a build description.

If you haven’t programmed with a purely functional language before, you can think of the Nix language as a kind of JSON but with simple functions. Packages are made of nested records that define properties, but you can also define functions to reduce duplication.

Functions are used in Nix to define common build procedures, so that many packages can share (for example) the rules used for compiling a Haskell program. There are also functions that modify other packages — say, takes a package and returns a variant with only statically linked binaries.

For another example of why it’s useful to define packages with a programming language, you could look at solc-versions.nix in the DappHub overlay, which defines many versions of the Solidity compiler.

Nixpkgs

Nix as a base layer is a very general build and installation tool. The repository of packages, Nixpkgs, is conceptually separate. It has more than 100,000 commits and is one of the most active projects on GitHub.

As an example of a Nix package in Nixpkgs you might look at the Z3
package definition which provides the SMT logic solver used by recent versions of Solidity. The package is defined in terms of the location and hash of its source code, which Nixpkgs dependencies are needed, and some simple modifications to the default build procedure.

Vendors can provide overlays based on Nixpkgs. For example, Mozilla has its own overlay with cutting edge versions of Rust, Servo, Firefox, and so on. The DappHub channel is also an overlay, pinned to a specific revision of Nixpkgs for complete reproducibility.

NixOS

There’s also a distribution fully based on Nix, and it’s pretty remarkable!

With NixOS, your entire system configuration is a package and all changes to your setup are both atomic and reversible. Indeed, the boot loader menu presents a choice of previous system configurations, timestamped and ready to launch.

If you’re scouting for a new distribution, give it a shot! Plenty of people and companies use it, and it has the 6 month release schedule with backports paradigm that we all love.

Plans for Nix and Dapp tools

With the infrastructure in place, we’ll go on to integrate Nix with the dapptoolkit.

The first use case is the ability to build and test with different Solidity compiler versions. This is already a one-liner with the nix-shell tool, but we want to make it as easy as possible.

We have more use cases in mind, like the ability to easily spin up a private gethtestnet with your dapp deployed. Could we maybe define Solidity packages themselves as Nix packages?

It’s both exciting and relieving to finally have a really good platform for defining software packages. Yow!

Thanks to all the Nix contributors and everyone who’s helped me figure out how to set up a custom channel.