A guide through The Wild Wild West of setting up a mono repo with TypeScript, Lerna and Yarn Workspaces

Auke ten Hoopen
Sep 8, 2020 · 11 min read

This is were the fun starts! Setting up a mono repo from scratch. We will be using TypeScript, Yarn workspaces, Lerna, and Jest. The release of the packages will be done with Github Actions.

In this part, I will explain step by step how to setup Lerna, Yarn Workspaces and Typescript in a mono repo. In the next article, we will focus on how to get Jest working. And the last part, we will handle the release of the packages with Github Actions.

You can find the full example and more in the template repo:

https://github.com/ahoopen/typescript-mono-repo

Lerna

What is Lerna and why do you need it?

Lerna is a tool that optimizes the workflow around managing multi-package repositories with git and npm.

So how does a mono repo with Lerna looks like?
There’s actually very little to it. You have a file structure that looks like this:

mono-repo/
package.json
packages/
package-a/
package.json
package-b/
package.json

Getting started

We will use Yarn as our package manager. Mainly because we are using Yarn Workspaces . So if you haven't installed it yet. Please do.

https://classic.yarnpkg.com/en/docs/install

Let’s start by installing Lerna as a dev dependency of your project

npx lerna init

This will create a lerna.json configuration file as well as a packages folder, so your folder should now look like this:

mono-repo
├── packages
├── README.MD
├── lerna.json
├── package.json

We still need some additional dependencies. So let's install them:

yarn add --dev typescript jest eslint

Now that all the dependencies are installed, let's continue with the Lerna.json

Lerna configuration

The lerna.json is where all the configuration happens for Lerna.

{
"packages": ["packages/*"],
"npmClientArgs": ["--no-lockfile"],
"version": "independent",
"command": {
"version": {
"ignoreChanges": ["*.md"],
"npmClient": "npm",
"message": "chore(release): publish"
},
"publish": {
"npmClient": "npm"
}
}
}
  • packages: An array of globs to use as package locations.
  • version At default, this is in fixed mode. Which keeps all packages versions in sync, but you can change this to independent if you’d like to manage package versions independent from one another.
  • command.version.ignoreChanges: an array of globs that won't be included in lerna changed/publish. Use this to prevent publishing a new version unnecessarily for changes, such as fixing a README.md typo.
  • command.version.message: a custom commit message when performing version updates for publication. See @lerna/version for more details.

With that out of the way, the next step is adding packages to our mono repo.

Create our first mono repo packages

Let’s create some packages in our repo. Create package/web and package/core. The content of the package should be as following:

├── src
│ ├── components
│ │ └── button.tsx
│ └── index.ts
├── package.json
├── tsconfig.json

The package.json should look like this:

{
"name": "@namespace/core",
"version": "1.0.0",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"repository": {
"type": "git",
"directory": "packages/core",
"url": "https://github.com/ahoopen/typescript-mono-repo.git"
},
"scripts": {
"clean": "rimraf dist",
"prepack": "yarn build",
"build": "yarn clean && yarn compile",
"compile": "tsc && cp \"./package.json\" ./dist/",
"test": "echo 'no test'",
"lint": "eslint \"./src/**/*.{ts,tsx}\" --max-warnings=0"
},
"dependencies": {
"react": "16.13.1"
}
}

And the tsconfig.json

{
"compilerOptions": {
"target": "es5",
"outDir": "./dist",
"module": "commonjs",
"jsx": "react",
"strict": true,
"allowSyntheticDefaultImports": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true
}
}

Running Lerna commands

{
"name": "root",
"private": true,
"scripts": {
"build": "lerna run build --stream",
"lint": "lerna run lint --stream --parallel",
"test": "yarn jest --coverage"
},

"devDependencies": {
"eslint": "7.7.0",
"jest": "26.4.2",
"lerna": "3.22.1",
"typescript": "4.0.2"
}
}

The lerna runcommand will run in every package of the mono repo. --stream flag will display the build output in the terminal and the --parallel flag will execute the command in parallel.

Caveat

because of sub-dependencies (packages a and b depend on c, d depends on a, b, c) we need to build packages accordingly, e.g. first c, second a and b, third d. That is why we can't use --parallel flag for lerna for build command.

Let's build our packages with Lerna. Run the following command in the root of our repository.

yarn build

If everything goes well, you will have the following output

lerna success run Ran npm script 'build' in 2 packages in 2.7s:
lerna success - core
lerna success - web

Optional: migrate an existing package to the mono repo

yarn lerna import ~/projects/my-npm-repo-package-1 — flatten

What’s great about Lerna is that it brings in all of the git commits along with it, such that for the new repo, the history looks like development has always been happening in this monorepo. This is crucial because you are going to need the git history when someone discovers a bug after moving to the monorepo.

Why Lerna alone is not enough

Lerna calls yarn/npm install for each package inside the project and then creates symlinks between the packages that refer to each other.

Being a wrapper of a package manager, Lerna can’t manipulate the contents of node_modules efficiently:

  • Lerna calls yarn install multiple times for each package which creates overhead because each package.json is considered independent and they can’t share dependencies with each other. This causes a lot of duplication for each node_modules folder which quite often uses the same third-party packages.
  • Lerna manually creates links between packages that refer to each other after the installation has finished. This introduces inconsistency inside node_modules that a package manager may not be aware of, so running yarn install from within a package may break the meta structure that Lerna manages.

These kinds of issues call for support of multi-package repositories directly in a package manager. Luckily there is one who supports this.

Introducing Yarn workspaces

Yarn Workspaces is a feature that allows users to install dependencies from multiple package.json files in subfolders of a single root package.json file, all in one go.

Yarn Workspaces enables faster, lighter installation by preventing package duplication across Workspaces. Yarn can also create symlinks between Workspaces that depend on each other and will ensure the consistency and correctness of all directories.

Let's enable Yarn workspaces in our mono repo. First, go to your lerna.json file and add the following fields to your configuration:

  • npmClient We set this to Yarn because we are going to use Yarn workspaces
  • useWorkspaces this will let Lerna know that we are using Yarn workspaces.
{
"packages": ["packages/*"],
"npmClient": "yarn",
"useWorkspaces": true,
"npmClientArgs": ["--no-lockfile"],
"version": "independent"
}

In our rootpackage.json we have to add this to enable yarn workspaces:

{
"name": "root",
"private": true,
"workspaces": {
"packages": ["packages/*"]
},
"scripts": {
"build": "lerna run build --stream",
"lint": "lerna run lint --stream --parallel",
"test": "yarn jest --coverage"
},
"devDependencies": {
"eslint": "7.7.0",
"jest": "26.4.2",
"is-ci": "2.0.0",
"rimraf": "3.0.2",
"lerna": "3.22.1",
"typescript": "4.0.2"
}
}

When running Lerna commands, it will now use Yarn Workspaces to bootstrap the project and also it will use package.json/workspaces field to find the packages instead of lerna.json/packages.

Adding TypeScript to the mix

Setting up Lerna and Yarn Workspaces is pretty straightforward. To set up TypeScript we have to configure a bit more because we have to transpile our code before publishing and this effects navigating the source code in our IDE.

We going to configure our mono repo in such a way that it will feel exactly as it would in a regular NPM package, meaning that Go to definition will work after a fresh clone.

Common compiler options

In the root of our mono repo we create a tsconfig.build.json . This will centralize our common compiler options. This way you can change a setting in one file rather than having to edit multiple files. All packages will extend this configuration file.

{
"compilerOptions": {
"target": "es5",
"outDir": "./dist",
"module": "commonjs",
"jsx": "react",
"strict": true,
"allowSyntheticDefaultImports": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true
}
}

Create packages/core , packages/utils and packages/web . (if you haven’t done already) An add atsconfig.json to each one. This tsconfig.json will extend our common compiler options.

{
"extends: "../../tsconfig.build.json",
"compilerOptions": {
"rootDir": "./src",
"outDir": "./dist"
},
"include": ["src/**/*"]
}

For each tsconfig, we have to declare the rootDir, outDir , include properties. They can’t be hoisted in the root config because they are resolved relative to the config they are in.

In the package.json of the package, we have to update the compile script that will build our package.

{
"name": "@namespace/core",
"version": "1.0.0",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"repository": {
"type": "git",
"directory": "packages/core",
"url": "https://github.com/ahoopen/typescript-mono-repo.git"
},
"scripts": {
"clean": "rimraf dist",
"prepack": "yarn build",
"build": "yarn clean && yarn compile",
"compile": "tsc --build && cp \"./package.json\" ./dist/",
"test": "echo 'no test'",
"lint": "eslint \"./src/**/*.{ts,tsx}\" --max-warnings=0"
},
"dependencies": {
"@namespace/utils": "1.0.0"
}
}

We build our package with the --build flag. This is important because we want to use theproject references from TypeScript. I’ll explain more about it further on

Enabling easy code navigation

By default, the TypeScript compiler will look for atsconfig.json. Our IDE will use this file to navigate through the project. Here we will map our package names to their source code in the monorepo.

Create a tsconfig.json in the root. This file will enable seamless navigation in our IDE across the mono repo. The configs need to be separate because the solution produces unintended behaviour when building packages.

{
"compilerOptions": {
"jsx": "react",
"baseUrl": "./packages",
"paths": {
"@namespace/core/*": ["core/src/*"],
"@namespace/utils/*": ["utils/src/*"],
"@namespace/web/*": ["web/src/*"]
}

}
}

Paths property is where the magic happens: it tells the TypeScript compiler that whenever a module tries to import another module from the mono repo it should resolve it from the packages folder. Specifically, it should point to that package’s src folder because that’s the folder that will be compiled when publishing.

The paths config should contain one entry per package, or if they have a common prefix like an organization, a glob that matches them all.

A big advantage of using path alias is that when you change a piece of code in one of the packages, the TypeScript compiler immediately picks it up without having to build all the packages.

Important caveat

If we try to build the corepackage (which depends on the utilspackage) in a fresh state (no dist folders anywhere) then it will fail and complain that it can’t find utils, even though Lerna symlinked it in its node_modules folder. This is because utils's package.json defines its entry point via the main field to point to dist/index. When the TypeScript compiler sees import '@namespace/utils' in the corepackage it will look for core/node_modules/@namespace/utils/dist/index.js and, because utilshasn’t been built (therefore there’s no dist/ folder in it), it will fail.

How can we solve this issue? This is where project references from TypeScript come into play.

Project references

Before project references, if you changed a package, you needed to re-compile it and also re-check any package depending on it, with no automatic detection of whether a change actually changed the types. If your packages needed to be built in a particular order, you were on your own as far as enforcing it. Furthermore, while tsc has had a watch mode since TypeScript's first release, before project references it wouldn't see changes in dependencies.

TypeScript will automatically compile its dependencies in, correct order and only if needed, before building and type checking the main project only if needed. This reduced our build times by about half on average and got rid of a really gross and unwieldy set of package.json scripts that we used to specify build order.

Referenced projects must have the composite setting enabled. This setting is needed to ensure TypeScript can quickly determine where to find the outputs of the referenced project.

Oke, let’s enable project references in our packages!

We have to modify the tsconfig.json of our packages. By setting “composite” to true, we let TypeScript know we want to use project references. This has to be enabled for each package. Furthermore, we have to include the project references. References is an array of paths of dependent packages. In this example, our package is dependent on the utils package.

{
"extends: "../../tsconfig.build.json",
"compilerOptions": {
"composite": true,
"rootDir": "./src",
"outDir": "./dist"
},
"references": [
{ "path": "../utils" }
],
"include": ["src/**/*"]
}

Unfortunately, we can’t place the references in the roottsconfig.build.json This means we need to add the references to each individual tsconfig per package.

We have to change the compile script in our package.json to let it work with project references. Instead of tsc we now have to do tsc --build . Running tsc --build will do the following:

  • Find all referenced projects
  • Detect if they are up-to-date
  • Build out-of-date projects in the correct order

We also need to update the clean script to remove the build cache for tsc — build otherwise, we can run into a situation where if the dist/ folder of a package is removed, but the package itself doesn’t contain any changes from last time, TypeScript will not rebuild it.

{
"name": "@namespace/core",
"version": "1.0.0",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"repository": {
"type": "git",
"directory": "packages/core",
"url": "https://github.com/ahoopen/typescript-mono-repo.git"
},
"scripts": {
"clean": "rimraf dist && rimraf tsconfig.tsbuildinfo",

"prepack": "yarn build",
"build": "yarn clean && yarn compile",
"compile": "tsc --build && cp \"./package.json\" ./dist/",
"test": "echo 'no test'",
"lint": "eslint \"./src/**/*.{ts,tsx}\" --max-warnings=0"
},
"dependencies": {
"@namespace/utils": "1.0.0"
}
}

As you can see, the process of adding new references is quite cumbersome. Especially if we have a lot more packages in your mono repo on which you depend.

Wouldn't it be great if we can automate this?

Auto configure project references

Instead of keeping the responsibility of keeping the project references up to date by the developer, we automate it. This way we know for sure that it is always up to date. And we take away an annoying task from the developer. A win-win situation!

Create aconfigure-references.js file at the root of mono repo with the following content:

We have to run the configure-reference script after every yarn install because a package might have a new dependency on one of the mono repo packages. Let's add the following script to our root package.json

{
"name": "root",
"private": true,
"scripts": {
"prepare": "node ./configure-references.js",

"build": "lerna run build --stream",
"lint": "lerna run lint --stream --parallel",
"test": "yarn jest --coverage"
},

"devDependencies": {
"eslint": "7.7.0",
"jest": "26.4.2",
"is-ci": "2.0.0",
"rimraf": "3.0.2",
"lerna": "3.22.1",
"typescript": "4.0.2"
}
}

After each yarn install the configure-reference script will be called and updates all references in all packages. Your project references will always be up to date :)

Conclusion

The first part is done! We can now easily navigate our mono repo. By making use of Yarn Workspaces, we enable faster, lighter installation by preventing package duplication across our repository. Last but not least, we have added TypeScript to our packages. But we are not there yet! Next steps will be, configuring testing in a mono repo and the releasing of packages.

The next article is all about testing. How to get Jest playing nice with our mono repo setup. So stay tuned!

AH Technology

Sharing our knowledge when building the world greatest Food & Tech company. #ahtechnology

AH Technology

Albert Heijn, part of Ahold Delhaize, is the number one food retailer in the Netherlands. We’ve been inspiring customers and building trust for more than 130 years. We making better food accessible to everyone — and making shopping fit the way people live today. #ahtechnology

Auke ten Hoopen

Written by

Lead Frontend developer @ Albert Heijn

AH Technology

Albert Heijn, part of Ahold Delhaize, is the number one food retailer in the Netherlands. We’ve been inspiring customers and building trust for more than 130 years. We making better food accessible to everyone — and making shopping fit the way people live today. #ahtechnology