Socks On Before Shoes — Smarter Way to Upgrade JS Dependencies
This article first appeared on xMatters.com.
As we work in more and more projects in a modular manner, we end up creating a number of packages which all have dependencies of their own. Updating and managing these dependencies becomes a problem after some time. Some factors that contribute to this are slow development cycles (combined with fast publishes in the JS world), avoiding scope creep, and testing burdens.
So we sometimes end up with packages which work just fine, but have some outdated dependencies we want to renovate.
There are a few solutions to this problem of updating package dependencies, but none in the wild were quite right for us. The biggest players in this space are Greenkeeper and Dependabot (now acquired by GitHub!). It’s no mystery why GitHub acquired Dependabot, as it’s very tightly integrated into GitHub itself. Greenkeeper is also a great solution and very tightly integrated with GitHub, but it’s not free for private repositories.
What to do if you’re not using GitHub at all? Or you have private packages whose dependencies you still would like to keep updated?
Well, until now, there weren’t many dependency manager options, which left most of us doing it manually.
How many Pull Requests to upgrade all dependencies?
Another issue that we’ve had with previous solutions is that, very cleverly, they do the bare minimum of upgrading a dependency, and leave the rest to your CI/CD pipelines. This is great for most cases; but in a world where time means money, you might have to think twice about having to wait for and merge 25 pull-requests that update 25 different dependencies.
Finally, instead of that evergreen-utopia, you might want to make conscious and well-tested upgrades that help keep your sanity in your day-to-day environment.
upgreat ups the dependency manager game
We developed upgreat to address some of these issues and make it easier for us to manage, monitor and schedule dependency upgrades for our internal projects, and are now releasing it to the wild in hopes that it will be useful for other developers and organizations.
upgreat aims to address some of these issues and operates in a different way. You plan your upgrade path, then execute it in one go. When it’s done doing its thing, you are left with a repository that has all its dependencies upgraded. Any upgrades that cause test or build failures are rolled back, and upgraded to the latest version for which the tests still pass. Finally, you get a nice report of what was upgraded, and get a list of changelogs for those dependencies that failed the upgrade.
Another nuance here is that you can upgrade all of your devDependencies without your tests failing once. Sure, upgrade all my build tooling and run unit tests afterwards, what could go wrong? upgreat runs build scripts if you’re updating devDependencies to save you from headaches.
In a way, upgreat embraces the “manual-ness” and the human element of undertaking this task, while making things easier for the maintainer(s) at every step.
Socks on before shoes
On a cursory look you could think that doing such an upgrade is super simple. Get the list of all outdated dependencies, then loop over them and upgrade each one to the latest one. Done!
Or is it?
This was our first approach to the problem, but it proves to be rather fragile and too naive for any sufficiently complex package. Lets take a simple app with Webpack for example:
// package.json
"file-loader": "0.11.1",
"webpack": "2.5.1",If you upgrade file-loader to 4.2.0 first, and then try to upgrade webpack to 4.41.2 afterwards, you will get into a state where your builds will be broken. Do you see why?
It’s because file-loader versions have peerDependency to webpack versions. You need to upgrade the libraries in the correct order to avoid running into issues. Just like in real life, if you put your shoes on first, and then try to put your socks on, you will likely fail.
upgreat’s solution to this problem is straightforward. Just like webpack, make, and a host of other projects, upgreat sorts the packages to be upgraded in topological order. This makes sure that any packages that others in the list depend on are upgraded first, and avoids any nasty collisions.
Topo-what?
Scary as it might sound, topological ordering is a straightforward concept. When presented with a list of dependencies and a flow, we want to find an ordering that will acknowledge the dependencies (like file-loader@4.2.0 depending on webpack@4.41.2).
Think of it as the order in which you would put on your socks and shoes. The astute among you might realize that there could be many different topological orderings for a given dependency graph. The good thing is, they are all valid and will work just fine.
Here are some ways you can put on your socks and shoes:
Left Sock -> Right Sock -> Left Shoe -> Right ShoeLeft Sock -> Right Sock -> Right Shoe -> Left ShoeLeft Sock -> Left Shoe -> Right Sock -> Right ShoeRight Sock -> Left Sock -> Right Shoe -> Left ShoeRight Sock -> Left Sock -> Left Shoe -> Right ShoeRight Sock -> Right Shoe -> Left Sock -> Left Shoe
and various permutations. Ultimately, the end state will be that you have worn socks and shoes on both feet. The order in which you do it (despite the heated arguments you might read on the internet about the right way of doing it) does not matter, as long as you don’t break the internal dependency between Sock -> Shoe.
Similarly for upgrading dependencies, the exact order of operations doesn’t make much difference, as long as you address the inter-dependencies.
Silver bullets and panaceas
I don’t think I need to tell you that there is no foolproof and fully automated way to upgrade dependencies of your projects. Unless you choose to live on the wild side and not write tests, but then you could just include package@latest until everything breaks down one day, including your sanity.
After trying a few of these approaches in xMatters, we’ve found this one works okay for our purposes. It automates the most grueling parts of the library upgrade process; but after all is said and done, it leaves it up to you to follow through and commit the changes. You can pretend you did it all by yourself, I won’t tell.
As a developer it feels great to be able to give back to the community who have basically provided for my livelihood over the years with amazing open source projects. I hope to be involved in more projects as we embrace open-source in xMatters and share some of our ideas and other projects with the community from which we all benefit greatly. Check out upgreat at https://github.com/xmatters/upgreat. I would love to hear your thoughts on your experiences doing library upgrades and opinions on upgreat.
