Monorepo setup with Lerna and Yarn workspaces

Photo by Austin Neill on Unsplash

For the last six months we’ve been building a node.js microservices framework for one of our customers. This framework is composed of many interdependent packages.

In this scenario, having one git repo for each package would be a burden. Too many npm links and too many projects to manage in our CI environment. Also new features may affect many packages so we have to deal with several related pull requests. It’s not likely that Github or Bitbucket gives us multi repo PR support anytime soon.

To overcome this shortcomings, in this project we went the monorepo way using Lerna and Yarn workspaces.

Lerna

Lerna is a tool that optimizes the workflow around managing multi-package repositories with git and npm.

Lerna tries to ease the management of npm links when dealing with multi package projects hosted in a single repository. It analyzes the packages and automatically creates the required npm links between them. It also handles execution of tasks across multiple packages and eases the pain of versioning and publishing.

It has its shortcomings but it’s worth using it. Big projects like Babel and Jest use Lerna.

Yarn workspaces

Workspaces are a new way to setup your package architecture (..) It allows you to setup multiple packages in such a way that you only need to run yarn install once to install all of them in a single pass.

Your dependencies can be linked together, which means that your workspaces can depend on one another while always using the most up-to-date code available. This is also a better mechanism than yarn link since it only affects your workspace tree rather than your whole system.

Yarn is in many ways much better than npm. We love Yarn and use it in all of our projects. You can use the workspaces feature with or without Lerna. When both projects are used together, Lerna delegates the dependencies management to Yarn.

The challenges

There are several limitations when using Lerna and Yarn workspaces together.

• Packages must declare its devDependencies locally if they want to use binaries in their npm scripts. Otherwise you need to use ../.. and may end having some path related problems.
• npm scripts should be executable using Lerna in the monorepo root and directly inside each package. This is necessary when dealing with scripts that run in watch mode (for example jest --watch).
• Lerna handles versioning and publishing but it’s not designed to work with private npm repositories.

Repo structure

This is our project structure:

├── node_modules
├── packages
│   ├── awesome
│   │   └── package.json
│   ├── awesome-module1
│   │   ├── src
│   │   ├── test
│   │   ├── babel.config.js
│   │   ├── eslint.config.js
│   │   ├── jest.config.js
│   │   ├── prettier.config.js
│   │   ├── package.json
│   │   └── task -> ../../scripts/task
│   ├── awesome-module2
│   │   └── ...
│   ├── ...
│   └── awesome-tools
│       ├── src
│       │   └── test-helpers.js
│       ├── babel.config.js
│       ├── eslint.config.js
│       ├── eslint.jest.config.js
│       ├── jest.config.js
│       ├── prettier.config.js
│       ├── package.json
│       └── task -> ../../scripts/task
├── scripts
│   ├── publish
│   └── task
├── lerna.json
└── package.json

• Yarn handles the dependencies.
• Lerna handles tasks that affect multiple packages (compile/test/lint all modules).
• One folder per package inside packages.
• All packages share the same structure.
• Each package defines only its runtime dependencies.
• All the tooling and devDependencies are shared and live in its own package.
• Each package contains the required configuration files for the tooling. Each file extends a common base configuration (we use Babel, Jest and ESlint + Prettier to compile, test and lint/prettify the code).
• Each package symlinks a common task script that defines how the different tools must be invoked.
• There is a “hub” package. It depends on all the other packages and allows easy usage of the framework (a single awesome dependency).
• All packages share the version number. We use lerna to update the version number in one fell swoop.
• Publication is handled by a custom publish script that will be used by the CI environment.

It’s easy to add new packages because they share the same structure. There also are plenty of extension points: any package can add something unique if it’s necessary (for example, its own linting rules).

Let’s see each part in greater detail.

Monorepo base: Lerna and Yarn

The root package.json looks like this:

And the lerna.json config file:

We setup the workspaces using the workspaces entry in package.json.

• We define tasks to clean/compile/test/lint all packages using Lerna.
• To update the version number we use the lerna publish command. In our scenario we do not allow Lerna to add commits or tags to the repo. We also avoid package publication.
• There is a check-packages task that will be used in the CI environment.
• There is also a publish-packages task for the CI. We will detect version number changes and publish the packages if needed.

Shared tools

Instead of polluting the root package.json with all the tooling we define another package to host the tools.

Notice that we will never define a devDependency to this package. We are abusing the workspaces here. Yarn will expand this packages’ dependencies in the root node_modules where other packages and tasks can find them.

Besides the dependencies, this package contains config files for each tool, and some shared code (like the test helpers). For example, this is our shared babel config:

Each package will rely on this shared config (more on this later).

The task script

The task script does two things:

• Defines the set of common tasks: clean, compile, test and lint. Each task uses the required tool forcing it to use the current package’s folder config files. It also fixes the base dir if needed (for example, jest needs this).
• Allows execution of any node_modules binary, in case the common tasks are not enough. For example a custom compilation command could be: ./task babel -d ./lib2 ./src2.

Package configuration

Each package just defines its runtime dependencies and a set of common scripts. The scripts delegate on the symlinked shared task script.

Each package also has their ownbabel.config.js, eslint.config.js, jest.config.js, and prettier.config.js. By default they just import the shared config exposed by the tools:

and if needed they can extend/override the config. It’s plain JS!

The hub package

This package is just a package.json that depends on all the other framework packages.

The publish script

To publish the packages we rely on our CI server. Using a custom script we’ll check each publishable package looking for the current version in the registry. If we pushed a new version to the repo, the module will be published.

No pain, no gain

It may seem simple now, but at the time we setup this project, there wasn’t much documentation. We had to do a lot of trial and error and evolve this setup to get to the point where we were really comfortable.

Using this as an example can help you setup your own monorepo. It should be easy to add the stuff that your project needs (maybe you use webpack or rollup, who knows). I have intentionally left out of this post our ESDoc configuration and how we generate one shared documentation for all the packages. Let’s leave it for a future post.