The package manager to rule them all

12 min readDec 31, 2022

It starts when you realize that you have outdated software detritus lurking on your system. You diligently tell your package manager to update everything, but it doesn’t. It can’t. It only manages one slice of your system.

Drowning in package managers

On Linux, your distribution of choice determines your package manager: apt, rpm, dnf, pacman, etc. On Mac, you might use homebrew or ports. But these system-level package managers barely scratch the surface, especially if you’re a power user.

Apps: On a Mac, you might have some things installed via the App Store, some installed via homebrew cask, and some apps installed via direct download or even compiled from source or via some Internet bash script (::shiver::). Linux is similar but with AppImage, Snap, and Flatpak instead of homebrew casks and the App Store.

Dev Utilities: If you use the command line very much, then you probably also find yourself installing things globally using language-specific installers like these:

npm install -g some-tool for node
cargo install some-tool for rust
pip install some-tool for python
gem install some-tool for ruby
go get -u some-tool for go

Dev Environments: But wait, you probably also have environment managers for different languages to manage the compiler and toolkit versions. These include utilities like nvm, rvm, rustup, pyenv, gvm, and so on. When you install a dev utility, like with npm install, you probably only installed that utility for the current dev environment, which means that tool will disappear if you change from Node 14 to Node 16 or whatever. Or worse, you’ll install it multiple times, one for each environment.

Plugins: Plugins are everywhere. I used oh-my-zsh to manage shell plugins, packer.nvim to manage my neovim plugins, and then some of those plugins were themselves package managers that would download and install things for treesitter and LSP and such. I had a manager for tmux plugins. My browser is effectively a package manager for plugins, too. Do you use VSCode? It’s a package manager for extensions and their dependencies, which it installs in a free-wheeling way.

The more you look, the more you find.

Dependency hell

This brings many problems, but the biggest is dependency hell. As you update some things, it breaks other things. The global environment makes it hard to sandbox different tools so they can work with their supported toolchains and libraries. So if you update one library, it fixes one thing while breaking another. In my Windows years, we called this DLL hell, but the problem generalizes across OSes. Sound familiar?

The update problem

It turns out, programmers make mistakes (what!!!???). And maybe you could learn to live with little quirks and bugs here and there, but at least some of those issues are security flaws. If you’re running old software, you’re vulnerable to something.

There are no two ways: you need to update often to keep your machine reasonably secure.

But of course, your software actually gets updated in many different ways.

For a while, I used a custom xbar bash script on Mac that would check for outdated items across the app store, homebrew, cargo, pip, and npm. It was pretty good and I was happy, but it still missed things.

I tried topgrade for a while (abandoned by its creator, it now has a maintained fork). It was better in some ways (it handled my editor and tmux plugins), but I also kept hitting issues with it. Yet even if it were flawless, I still wouldn’t be in my happy place.

And what about the configs?

It turns out all updates are not merely minor bug fixes. Some involve breaking changes to APIs and config file formats and such. So your perfectly dialed-in system with everything humming along in perfect tune can very quickly crap the bed on an update.

I’m a die-hard vim user. I know VSCode is pretty, but I can FLY in vim (actually, neovim for me these days). I’ve left it many times, and I always come back.

But damn it, it takes work to maintain a (neo)vim config, and plugins change often. (And yes, it’s possible that my awesome productivity when using vim is counterbalanced by the time spent tweaking configs — but let’s not talk about that.)

So some updates would go by with everything staying happy, and other times, things would explode because vim itself updated or some plugin did. Autocomplete would break. Notices on startup would tell me about deprecated options. Suddenly two plugins were competing to do the same thing because one added new functionality.

And with (neo)vim and elsewhere, it gradually dawned on me that my config files were intimately tied to specific software versions. But those config files were also global mutable changing things.

It turns out that having a zillion different package managers and config files all playing in a single global environment was hell.

Welcome to my happy place

What if one package manager could manage everything, including those vim plugins, tmux plugins, npm packages, and so forth?

What if utilities could be sandboxed, so their dependencies didn’t screw with the dependencies of other utilities and apps?

What if my dev environment for one project could be nicely isolated to that project, and changes to it didn’t blow up other (often older and less visited) projects?

What if my setup could be fully declarative and used to deploy my things to new computers quickly?

What if my configurations were managed and upgraded the same way as my software and versions of my configs were tied to versions of the software those configs controlled?

Welcome to my happy place where all of my dreams and what-ifs comes true.

I’ve become a super fan of Nix.

I should pause and note that my primary machine is a Macbook Pro, but I also manage an iMac, a Linux framework laptop, and a few Linux servers. And I’ve moved all of them to Nix.

For Linux, that means that I run NixOS, though when I started, I was content to just use Nix as an alternate package manager — and that works fine. On the Mac, I also initially just used Nix as a more traditional package manager, but I now use nix-darwin to control more of the system (like fonts). But you probably want to start as I did and layer Nix in as an adjunct until you’re ready to go the whole hog. (What is that expression anyway? A butchering reference. Bah.)

I use Nix to manage 99% of the software on my machines. Sadly, there are still edge cases, but I’m gradually eliminating them. And they’re mainly on Mac.

I never install things with npm install or cargo install anymore. It can be a hassle, but if nixpkgs doesn’t have what I need, I write my own recipe (“derivation”). And that gets easier with practice.

And look, nixpkgs usually has what I need. It has far more, and more up-to-date packages than any other package manager:

It has recipes for building packages that are often cross-platform, but even better, they have binary caches, too, so you can usually speed through installs.

Using Nix

You can use Nix just like you use homebrew or apt or rpm or whatever and install things into your environment at will. Most Nix tutorials teach you how to do exactly that. But I hate that. You lose half the value out the gate by just installing stuff willy-nilly.

To me, the real beauty of Nix is the ability to declaratively describe your entire system using text files.

My config files live in git and specify my apps, their dependencies, and my associated configs. Everything is installed and updated together. I try to discover when a config is broken and fix it before checking in the updated package versions (the new lock file). Then I can recreate any version of my system anywhere.

Experiments and Rollbacks

So say you’re on Linux, and you want to give Wayland a try. Cool. Change a couple of lines in your config. Maybe do it in a branch. Run that config and see if you like it or if everything fails miserably because you have the wrong graphics card or whatever. And if you find that some of your favorite apps just don’t play nicely with Wayland, you can trivially go back.

But that’s not all, because with NixOS and Darwin-Nix, you get system “generations” that stick around for as long as you want. This includes everything from the kernel to your X windows version to whatever. So if you update and reboot and your screen is unreadable, you’re in trouble because something — you don’t necessarily know what — broke your system. If you’re on NixOS, just reboot and select a previous generation to boot to. Voila, you’re back to a working state and can deal with the upgrade issues when you have more time.

If you’re not using NixOS, you can still roll back with a command. No need to rebuild or redownload — everything is already built and in place. You just select different parts of your /nix/store to be active. Delete old generations and free up space when you feel confident you don’t need them.

Isolation

Nix isolates everything. It’s pretty incredible. If one app needs OpenSSL 2.2 while another needs 2.4, that’s no problem because everything is sandboxed. App A won’t interfere with App B. If they use the same dependency, it will be shared, but if not, cool, they won’t conflict with each other. It’s magic. Oh, and that applies to how something is built, too; if there are feature flags that need to be enabled for one app but disabled for another, it doesn’t matter.

How it works (abbreviated)

The way it works is this: all libraries and binaries are built and stored in their own /nix/store/long-hash-name folder where the hash is over all of the build inputs, including versions (down to the specific git commits), build parameters, dependencies, and their versions and inputs, etc. That hash represents something that’s completely reproducible and the exact output given a series of inputs so you know if you already have the artifacts you need.

The built packages are then selectively brought into your environment by modifying your $PATH or $LD_LIBRARY_PATH. So binary X always sees precisely the versions of its dependencies that it needs, and it doesn’t matter if binary Y needs something else. In some cases, there are wrappers that set up a specific environment (set of available binaries and such) before running a program.

Behind the scenes, build scripts sometimes need to patch hard-coded paths and use other tricks to accomplish all this. In the end, though, the result is beautiful.

Dev environments

It gets even better in development workflows because Nix replaces those environment managers. It is an environment manager. No need for nvm or rustup or whatever. Each of your projects can declare what it needs, and then you can load just those things into your environment when you’re in that project directory.

And you can make it cross-platform. So if you have a bash script that relies on GNU grep and breaks on Macs, well, you can add gnugrep to the environment for that project so that everyone executes things with the same dependencies.

Now, this isn’t as awesome if the rest of your team isn’t also bought in since everyone will have to do some setup to make it work right. But on the flip side, if you get everyone on board, you’ll never have to remember to call nvm again. And you’ll be done with the “it works fine for me” crap where you have no idea what’s problematically different between your global environments and your coworkers’.

Flakes (neovim example)

You can use Nix to manage your VSCode setup, including configs and plugins. I have a config for that, but I rarely use it. I prefer neovim. It’s a steeper learning curve, but you can’t beat the productivity.

One confusing thing with Nix is that there’s the old way of doing things and the new way of doing things. The new way uses something called “flakes,” and the way to understand these is just that you can scope a flake to a project or to your system as you like. A flake specifies inputs and produces a flake.lock file that pins those dependencies (or the state of the nixpkgs repo, which indirectly pins the dependencies).

I’m on Team Flakes because it allows for composability and separation of concerns in ways that Old School Nix doesn’t.

And there’s a neat trick you can do with flakes where you can point to something on the Internet and just run it, temporarily, without installing it into your environment.

A while ago, I was helping a coworker get their NixOS config going. So I fired up a VM to run and test their config. But oh lord, was I lost without my comfortable shell, familiar tooling, and, most of all, my editor. This spurred me to extract parts of my system configuration into flakes so I could selectively use them from anywhere.

For example, while I can run nix-shell -p neovim to install nvim in a temporary shell and then use that to edit files, that comes with bare bones. I didn’t get my Nix language linter and formatter, and I didn’t have my nice autocomplete plugins.

So I set up a dedicated repo, so now I can just do: nix run github:zmre/pwnvim and it will run nvim with all the configs and dependencies I want without putting anything into that VM’s ~/.config or “installing” anything. I get everything I need, like LSP servers, code linters, formatters, the ripgrep tool, fzy, and my favorite plugins with a config just works.

Bliss.

Alternatives?

As a point of comparison, if you go try out some prepackaged neovim config like the awesome LunarVim (or AstroNvim or Nyoom.nvim or whatever), you’ll have a lot of installing of things to do, and if anything you install is newer or older than what the maintainers are using, there’s a reasonable chance that something that should just work will break.

Installing LunarVim today requires you to run a bash script that pollutes your system with config files and calls things like npm install and cargo install. And you have to have npm and cargo in your global environment in the first place, along with git, make, pip, python, npm, node, and more, to even get started. They don’t specify which versions.

That’s not a dig on them. Without Nix, I don’t know of another way for them to package a more reliable, reproducible, and comprehensive environment for you.

My config — which I’m not building to be generic for others at this point (should I?)— is self-contained and comes with all of its own dependencies. If something isn’t working for you, it’s not because your environment is different from mine but because I screwed up on something, and it’s screwed up for me, too.

The Nix dark side

Nix isn’t for everyone. It isn’t particularly easy to use or understand. It’s not too hard if you use it like other package managers, but then you lose much of the power.

Unfortunately, you probably need some time to dedicate to learning it and experimenting with it, and you’ll need to strap on your nerd pants for that.

I wish learning Nix were easier. Part of the problem is that Nix is powerful, and has many ways to use it. That confuses things tremendously.

The Nix language doesn’t help things, either. It’s hard to read (wait, that’s a function call? and it’s returning a function? how do you know?). It isn’t intuitive. Error messages are often misleading, which makes it hard to debug. There’s no intellisense to help you. And frankly, understanding how to package something often gets tangled up with the day-to-day of just using what’s already there for you.

There also isn’t a canonical starting point. Instead, there are dozens of starting approaches that are hard to sift through when you’re new, which can be daunting. (For what it’s worth, writing this blog spurred me to do a video walkthrough showing an install and basic config on MacOS.)

It’s worth it

But if you can conquer and overcome the difficult bits, damn, it feels great to have a fully reproducible system that is tracked, nicely upgradeable, and stable.

Ditch the 1,001 package and environment and plugin managers and replace them with the one package manager to rule them all: Nix.