Node.js Microservice Containers with Docker
Microservice Containers with Increased Productivity
Using Docker brings plenty of benefits to microservice projects. Continuous deployment, immutable deployment and consistency between development and production environments eliminate lots of the problems you might find using other approaches.
I have seen quite a few approaches to microservice deployment using Docker, primarily with Node.js. Most of them have a separate code repository per microservice, each with its own dependencies and build tooling. If you’re scaling to tens or hundreds of microservices in a large, established organisation, this might work out. If you’re working as part of a small team in a startup that wants to move fast, isolating each service’s codebase like this is not helping you to be productive. Here’s a list of some of the things I’d prefer not to spend my time doing.
- Manually updating a database driver version (or any other dependency version) in ten or more repositories
- Waiting minutes for my Docker image to build as it installs all NPM dependencies from scratch. (.dockerignore should prevent using any pre-built node_modules tree)
- Waiting minutes for docker push and docker pull to happen because my applications have hundreds of megabytes of frequently changing layers with minor differences to the previous version
If you’re making many changes to many applications on a daily basis, the waiting time is compounded as images are built and transferred in development, CI and deployment. It eats your time, slows you down and distracts you. It is well worth it to invest some time in shrinking build turnaround times to the optimal, minimal value.
Most Node microservice containers will take a base image from the DockerHub library, like node or node-slim. Take this a step further. Find the common, stable parts that you employ across the majority of your containers and push them into a base layer. It can start with little effort. This case is about Node.js but can be applied to any runtime. I have used it to reduce application containers from producing 3–400MB layers to layers less than 1MB in size.
Bundle dependencies in their own module
For a Node microservice application, you might well be using Seneca.js, Hapi or Express, Lodash and Async in most of them. Create a module with all of these as dependencies. Then, expose them to other modules using a method of your choice. With NPM, it is not common to refer to sub-dependencies so you must explicitly expose what your base module requires. The following deps module is one approach to this.
Now, your application dependencies can load the common module using syntax like deps.lodash. Include your base module as a dependency in all microservices, avoiding the repeated requires. If you later need to bump a dependency version for any microservice, you have a choice in how to do it. Change it in a new version of the base module or in the project repository itself.
Turn your base module into a base image
A Docker base image allows you to specify a FROM line in your application Dockerfile that provides most of what you need to run your container. You might typically use something like ubuntu or node-slim as your base. Building bases on top of bases extends the possibility for reuse and speeds things up considerably.
Here’s how a base image for our Node microservice container base might look.
Starting with node-slim, we install our base Node module into the image’s root filesystem in the node-seneca-base directory. Then we run npm link from that directory. This is the very simple bit of magic that allows us to trick the system into reusing modules already in our base image.
Moving on to the second part of the Dockerfile, the ONBUILD steps are executed only in Docker builds that use this base image in their FROM line. Instead of running npm install to get all dependencies in place, we first npm link back to the common base node module already built in our base image. This does little more than putting in place a symbolic link to the module in the base layer. When we then run npm install for our microservice container, the base module is already there and does not have to be installed again. Only the few, service-specific dependencies are put in place at that point.
Building the microservice container
The microservice container build becomes quick, simple and short. The Dockerfile for each microservice becomes minimal:
Few optimisations come without some drawbacks. You need to ensure you have a clear policy for what belongs in the base, how you update dependencies, run your builds and track version changes to minimise any additional dependency management burden. There is no obligation to keep all your microservices on the same base version, of course.
I haven’t tried this approach with NPM 3 but don’t see why it wouldn’t work as outlined.
Beyond base dependencies
Depending on your approach to Node modules, you may like to add more code into your base. Application configuration, startup and utilities could all belong there quite happily. The main consideration is avoiding frequent updates to the base that reduce the benefit of having a stable, reusable common base layer. If that is achieved, you should end up with faster build turnaround, simple, quick deployments and happier, more productive teams.
An example base image with sample microservice containers is available in GitHub at https://github.com/eoinsha/node-seneca-base with the corresponding Docker base image on DockerHub at https://hub.docker.com/r/eoinsha/node-seneca-base/