A truly reproducible scientific paper?
The current situation
One pillar of the scientific method is reproducibility, that is, being able to redo an experiment and get the same result. Sometimes this is very hard to accomplish. For example, if you’re doing field work, you might end up collecting some very rare samples or observe a rare natural event. If you work in the lab, tracking all the variables that go into an experiment and might influence it (e.g. the way a person pipettes or mixes reagents) is similar to following your mom’s recipes… most of the times it doesn’t give you the same amazing dish.
However, when analyzing data on a computer you would expect to have control of all the variables and be able to re-run everything you did at the touch of a button. Right? Well… it’s complicated. There are so many software layers with intricate dependencies relying on very specific versions to work that it’s very hard to keep track of everything and very unlikely to expect that old code will run in a new environment (operating system and software stack).
Virtual machines and containers only partially solve the problem. They allow you to re-run an entire software stack easily on another computer, but that does not mean you know exactly what is going on inside that virtual machine image, how was it created, or how the binaries inside were compiled. In many cases, you’re just ignoring the problem and relying on a black box. Then you start relying on multiple VMs and containers to run ONE experiment, with very specific images versions inter-dependencies, and you’re back to square one (possibly in a worse situation).
For more details, I recommend checking C. Titus Brown recent blog post about this problem “How I learned to stop worrying and love the coming archivability crisis in scientific software” and for a funny and historical perspective of how we got here, check out Joe Armstrong’s talk “The Mess We’re In”.
Is there a solution?
I’m going to share my current best approach to this problem. It’s far from perfect, so it would be interesting to know if someone has come up with something better. I think it’s a good enough solution for reproducibility and reusability.
At the programming language level
Currently I use each programming language package manager to keep track of specific versions of libraries used. Most of the time the best tools (i.e. less buggy) to install libraries for a programming language are the ones that the language uses (
pipfor Python, etc). So, relying on another package manager to do that job (i.e. the OS package manager), in my experience, gives you more trouble than it’s worth. However, sometimes this can have drawbacks for reproducibility (e.g. the left-pad crisis in Node) and we need to be careful.
Whenever possible, we should include the source code of our dependencies in our project and not rely on external sources. In a language like R, versioning and snapshotting the dependencies source code might be difficult, but there are solutions like Packrat. In Node.JS you can easily manage your dependencies with a
npm shrinkwrap. In Python you can do
pip freeze > requirements.txt. For both, there are ways to host your own local repositories.
At the Operating System level
Instead of relying on the OS packages manager (e.g.
rpmfor Fedora) to keep track of the languages themselves and all the other dependencies (e.g. some languages require specific versions of Fortran, GCC compiler, etc) I use the Nix package manager.
Nix is a functional package manager that stores packages usually in the directory
/nix. Each package has its own unique subdirectory, such as
which results from hashing the package’s build dependency graph. Don’t worry, you won’t have to deal directly with these paths to use the tools you install. But this will allow you to have deterministic installs and use multiple versions of the same tools without any conflict. And all the dependencies are isolated inside
/nix, so whatever you install will not rely on the host system (unless you have some dependency at the Kernel or Firmware level, but you’ll have the same problem with Docker containers).
Nix works on Linux, Mac, and latest Windows 10 Insiders preview’s bash.
How to tie it all in a reproducible paper
In the next blog post I will show you a practical example of what this looks like, stay tuned!