For the last six months we’ve been building a node.js microservices framework for one of our customers. This framework is composed of many interdependent packages.
In this scenario, having one git repo for each package would be a burden. Too many npm links and too many projects to manage in our CI environment. Also new features may affect many packages so we have to deal with several related pull requests. It’s not likely that Github or Bitbucket gives us multi repo PR support anytime soon.
Lerna is a tool that optimizes the workflow around managing multi-package repositories with git and npm.
Lerna tries to ease the management of npm links when dealing with multi package projects hosted in a single repository. It analyzes the packages and automatically creates the required npm links between them. It also handles execution of tasks across multiple packages and eases the pain of versioning and publishing.
Workspaces are a new way to setup your package architecture (..) It allows you to setup multiple packages in such a way that you only need to run
yarn installonce to install all of them in a single pass.
Your dependencies can be linked together, which means that your workspaces can depend on one another while always using the most up-to-date code available. This is also a better mechanism than
yarn linksince it only affects your workspace tree rather than your whole system.
Yarn is in many ways much better than npm. We love Yarn and use it in all of our projects. You can use the workspaces feature with or without Lerna. When both projects are used together, Lerna delegates the dependencies management to Yarn.
There are several limitations when using Lerna and Yarn workspaces together.
- Packages must declare its
devDependencieslocally if they want to use binaries in their
npm scripts. Otherwise you need to use
../..and may end having some path related problems.
npm scriptsshould be executable using Lerna in the monorepo root and directly inside each package. This is necessary when dealing with scripts that run in watch mode (for example
- Lerna handles versioning and publishing but it’s not designed to work with private npm repositories.
This is our project structure:
│ ├── awesome
│ │ └── package.json
│ ├── awesome-module1
│ │ ├── src
│ │ ├── test
│ │ ├── babel.config.js
│ │ ├── eslint.config.js
│ │ ├── jest.config.js
│ │ ├── prettier.config.js
│ │ ├── package.json
│ │ └── task -> ../../scripts/task
│ ├── awesome-module2
│ │ └── ...
│ ├── ...
│ └── awesome-tools
│ ├── src
│ │ └── test-helpers.js
│ ├── babel.config.js
│ ├── eslint.config.js
│ ├── eslint.jest.config.js
│ ├── jest.config.js
│ ├── prettier.config.js
│ ├── package.json
│ └── task -> ../../scripts/task
│ ├── publish
│ └── task
- Yarn handles the dependencies.
- Lerna handles tasks that affect multiple packages (compile/test/lint all modules).
- One folder per package inside
- All packages share the same structure.
- Each package defines only its runtime
- All the tooling and
devDependenciesare shared and live in its own package.
- Each package contains the required configuration files for the tooling. Each file extends a common base configuration (we use Babel, Jest and ESlint + Prettier to compile, test and lint/prettify the code).
- Each package symlinks a common
taskscript that defines how the different tools must be invoked.
- There is a “hub” package. It depends on all the other packages and allows easy usage of the framework (a single
- All packages share the version number. We use lerna to update the version number in one fell swoop.
- Publication is handled by a custom
publishscript that will be used by the CI environment.
It’s easy to add new packages because they share the same structure. There also are plenty of extension points: any package can add something unique if it’s necessary (for example, its own linting rules).
Let’s see each part in greater detail.
Monorepo base: Lerna and Yarn
package.json looks like this:
lerna.json config file:
We setup the workspaces using the
workspaces entry in
- We define tasks to
lintall packages using Lerna.
- To update the version number we use the
lerna publishcommand. In our scenario we do not allow Lerna to add commits or tags to the repo. We also avoid package publication.
- There is a
check-packagestask that will be used in the CI environment.
- There is also a
publish-packagestask for the CI. We will detect version number changes and publish the packages if needed.
Instead of polluting the root
package.json with all the tooling we define another package to host the tools.
Notice that we will never define a
devDependency to this package. We are abusing the workspaces here. Yarn will expand this packages’ dependencies in the root
node_modules where other packages and tasks can find them.
Besides the dependencies, this package contains config files for each tool, and some shared code (like the test helpers). For example, this is our shared babel config:
Each package will rely on this shared config (more on this later).
The task script does two things:
- Defines the set of common tasks:
lint. Each task uses the required tool forcing it to use the current package’s folder config files. It also fixes the base dir if needed (for example, jest needs this).
- Allows execution of any
node_modulesbinary, in case the common tasks are not enough. For example a custom compilation command could be:
./task babel -d ./lib2 ./src2.
Each package just defines its runtime
dependencies and a set of common
scripts. The scripts delegate on the symlinked shared
Each package also has their own
prettier.config.js. By default they just import the shared config exposed by the tools:
and if needed they can extend/override the config. It’s plain JS!
The hub package
This package is just a
package.json that depends on all the other framework packages.
The publish script
To publish the packages we rely on our CI server. Using a custom script we’ll check each publishable package looking for the current version in the registry. If we pushed a new version to the repo, the module will be published.
No pain, no gain
It may seem simple now, but at the time we setup this project, there wasn’t much documentation. We had to do a lot of trial and error and evolve this setup to get to the point where we were really comfortable.
Using this as an example can help you setup your own monorepo. It should be easy to add the stuff that your project needs (maybe you use webpack or rollup, who knows). I have intentionally left out of this post our ESDoc configuration and how we generate one shared documentation for all the packages. Let’s leave it for a future post.