Building a Resilient Continuous Deployment System
When was the last time a single developer did something really small and it made big impact? You can list them out or you can read it on Verge, BusinessInsider and on TheRegister. In a nutshell, Azer decided to unpublish left-pad, an 11-line NodeJS module on NPM, and it broke thousands of other NPM module installations that depended on it.
The impact of the absence of left-pad was felt mainly by services having a continuous integration or a continuous deployment system. I can but imagine the countless hours developers had to spend to patch up their CI/CD to prevent or fix a service outage.
The fact remains that our endeavour to deliver software quick and dirty is what has left us vulnerable. Vulnerable to a point where it is both disconcerting as well as lame
Did this happen to you?
It is not an uncommon practice for node driven servers to perform an `npm install` before deployment. During the unavailability of the said modules of Azer, a service could have had a scale-up triggered and failed to add new servers to the cluster as `npm install` on them failed. It could have also been that a service had an upgrade scheduled and it did not happen owing to the missing packages.
The above two are clearly the examples of infrastructure done in a haste. Pulling packages directly from NPM before deployments surely exposes one to these idiosyncrasies and beyond:
- If NPM goes down, you may go down.
- If someone unpublishes a dependency, you may go down.
- If someone publishes a buggy patch to a dependency, you may go down.
Those are not pretty things to go down for. We can geek-joke on this with left-pad.io, require-from-twitter or canibekikked, but they attempt to solve the issue from a different (and humorous) angle.
What is the real issue?
As you can read above, the blame game starts with NPM, moves on to the nature of JavaScript and finally rests with the abysmal state of the human race — more on Reddit and HackerNews.
Apart from the fact that NPM should now work on collaborating with organisations hosting package mirrors and prevent unpublishing of modules, the real issue lies with how we do not think twice before coupling the fate of our CI/CD systems with uptime of other services. It is not stupid to spend some practical amount of time to build failsafes — how else do you think we will build FTL drives?
The real solution…
People will not stop from from publishing and consuming single-line Node modules. This is an organic growth of the NodeJS ecosystem, which is in some way a novelty that many non-JavaScript developers secretly envy.
The trick is to plug the hole at the point of delivery.
- Lock your NPM module versions
- Move the process of fetching modules from deploy time to build time
- Setup a fallback layer between NPM and your services
Add a “packaging” step to your workflow
To begin, you should start by adding (if not already) a “package” or step in your deployment workflow. This step sits between developers (or CI) committing code and the source made available to server clusters.
This type of CD workflow decouples the CI and CD and interfaces then with a simple build artefact server. With the build store in place, builds can now be packaged and the above three safeguards be implemented.
The build store could be a simple S3 bucket or could be your own SFTP server or could be a container service such as private Docker registry or Amazon Container Service.
Locking your NPM modules
The first thing we can start is with shrinkwrapping your npm packages. By doing so, you recursively freeze the versions of the packages that are installed. This ensures that changes to indirect dependencies do not get into your way of development and deployments.
This simple solution would save you countless debugging hours that would have otherwise creeped in unannounced at the most unfortunate moments — such as the one with one senior engineer at SoundHound.
Package your modules with source
Once you have your modules shrink-wrapped in your packaging step, you can then move on to converting the dependencies to bundled dependencies. This can be done by running a script something similar to the following:
Once the package.json has been updated in the build step, we may simply use the `npm pack` to create a compressed file with relevant dependencies included in it. This can then be uploaded to the build store server we previously discussed.
If you are in a docker (container) based workflow, you could simply do an `npm install` at this point in a container and push the same to a private container registry. No point pushing it to docker hub as hard coupling with external service is exactly what we are avoiding.
With dependencies bundled, extracting this package and running `npm install` on it causes npm to use the modules first from node_modules folder and not ping the registry.
Host your own private NPM
In the event that having a build store is not favourable or if you have relatively large number of projects where managing build store does not have enough returns, you could try hosting your own private NPM registry.
There are couple of such services that work flawlessly out of the box
- sinopia (free and requires a bit more attention to make it fault tolerant)
- npm onsite (paid and is still under active pre-production development)
These servers can be configured to “cache” package installation requests and serve it from cache instead of NPM registry. No more brownouts due to npm outages.
PS: You should also have a look at npm-cli-login module that makes it easier for CI systems to login to private registries.
“The farther back you can look, the farther forward you are likely to see.”
~ Winston Churchill
—
Originally published shamasis.net
