Do we really need to use package managers?

I’ve become pretty skeptical of using package managers within organizations. At Google and Foursquare, we had monolithic main repositories, and checked third-party code directly into the repository (apparently Facebook does this as well). This way, when you cloned, you always got all the code you needed to get up-and-running: no extra flags, no broken, half-cloned repositories. And when you made a change, you would know exactly which downstream dependencies you were about to break.

Asana has historically had the same practice, until recently. Our desire to have more open-source code led us to adopt more community-standard practices, such as using multiple inter-dependent repos, using semantic versioning, and letting NPM download our dependencies instead of checking them into git. And so far, with the exception of our code looking more like other open-source code, things seem worse.

There are more build breaks, and builds are less repeatable between different machines. Bad semantic version updates means that code that was working yesterday does not work today, because I ran npm install. Breaking changes can take days or even weeks to make it to downstream dependencies, creating huge, headache-inducing merges. And if the NPM registry is down, Travis can no longer build, since it can’t fetch dependencies, even if it can fetch the code from GitHub.

Similarly to how git submodules introduce single points of failure, NPM packages do as well. In fact, downloading artifacts of any kind — be it git submodules, NPM packages, JARs from Maven or Ivy — introduces single points of failure into a clone or build process. If any of these systems is unavailable, your build will fail.

Checking these artifacts into your repository reduces your build’s vulnerable surface area, by reducing your dependencies on the Internet. And if you hate the idea of checking binaries into git, you can try git lfs or git annex, etc. (a dependency on GitHub and S3 seems strictly better than a dependency on GitHub, NPM, Maven, Ivy, etc.).

Fetching dependencies through a separate channel feels suspiciously similar to dynamic linking—artificially loosening versioning constraints between pieces of code in ways that can affect its correctness. But it’s also unclear how to play nicely within the open-source ecosystem without using the same technologies as other open-source projects. Google had the (now seemingly unmaintained) make-open-easy tool, but it was still (ironically) complex to add to a workflow.

If you know of an organization that has a pain-free, open-source-friendly workflow, let me know.