Migrating to a Monorepo Using NPM 7 Workspaces

A practical guide for developers to refactor and configure an existing repo to a Monorepo using npm 7 workspaces, webpack, babel, jest, and typescript

Photo by Ferenc Almasi on Unsplash

If you are one of the early birds using npm7 and looking to use their new workspaces feature, you probably noticed that there are not too many documentations as of today. For example, the only up-to-date documentation by npm is their RFC, and that’s why I’m writing this documentation. You definitely should read the RFC first.

I will demonstrate the process of moving an existing repo to a Monorepo on an app that its main dependencies are: styled-components, react, redux, webpack, babel, docker, typescript, eslint and jest.

This is an advanced guide, we will not get into these dependencies details, there are enough tutorials and How-to about them, rather we will focus on how to configure them correctly to fit a monorepo structure.

Why NPM 7?

That’s a great question. there are many solutions today for monorepo, and you definitely should explore all of them. just as a quick note, lerna tool is not updated often these days, looking for maintainers, and has many open issues, also it’s not the best solution in case you want to focus on building apps rather than libs. Nx on the other hand, is enforcing a specific project structure with specific dependencies and I found it very hard to customize, also you will find that it’s difficult to control packages versioning. yarn is a great option, but we already using npm . Also, npm declared that their main focus will be on improving workspaces in the upcoming releases. Indeed, npm or yarn are not completely alternative to tools such as lerna but they give you the basics you actually need like installing dependencies across child packages and packages symlink.

The first thing you should do is opt-in to npm7 using: npm i -g npm@7 or install Node.js 15 with Latest Features.

Restructure our current directory

Once npm7 is installed, we will start with restructuring the repository folders to fit Monorepo’s structure.

To begin, let’s define the initial state of our repo:

- .gitignore
- .storybook
- public
- scripts
- src
  - components
  - containers
  - actions
  - reducers
  - index.html
  .
  .
- .babelrc
- .eslintrc.js
- Dockerfile
- jest.config.js
- jest.init.js
- package.json
- tsconfig.json
- webpack.config.js

Our new structure will contain two main folders, apps and packages. The apps folder will contain applications that we will probably not publish to the external repository, while packages will contain modules that we will publish into npm registry / github packages. You do not have to publish them at the first stage, or at all, and decide to always work with the latest version, yet if you have another repository that will need to consume one of the packages (for any reason), then publishing is the way to go.

We want to change the current directory structure to the following. Make sure to use git mv while changing files locations to preserve files history:

- .gitignore
- apps
  - my-app
    - Dockerfile
    - docker
    - jest.config.js
    - jest.init.js
    - public
    - scripts
    - src
      - components
      - containers
      - actions
      - reducers
      - index.html
      .
      .
    - tsconfig.json 
    - webpack.config.js
    - package.json
- tsconfig.json
- .babelrc
- .eslintrc.js
- webpack.base.config.js
- packages

Everything related to the development, we will keep on the root level, to enforce the development standards across all apps and packages, that also means we should have most of our devDependencies only within the root package.json. This is a common practice in a Monorepo environment.

Create a root package.json file

Let’s also create a root package file with the following content :

{
  "name": "monorepo",
  "scripts": {
    "my-app:eslint": "eslint './apps/my-app/src/**/*.{jsx,js,tsx,ts}'"
    ....
  },
  "dependencies": {
  },
  "devDependencies": {
    ...
  },
  "workspaces": [
    "./packages/*",
    "./apps/*"
  ]
}

The main difference in this package.json file is that we have a workspaces new property. That tells npm where to look for workspaces. A workspace is a folder with apackage.json file in it.

we also created my-app:eslint script, to run eslint from the root folder but only under the scope of the app folder, this could also be helpful later for CI-CD.

And just like that, we have officially moved to a Monorepo.

Photo by Razvan Chisu on Unsplash

Share packages

The main advantage of monorepo is shared libraries, in our case we want to be able to use the components in different apps, so what we actually want to do is to create a shared library out of our components folder that we will be able to use across our many apps.

In order to do so, we are going to move the content of the components folder from the app folder into the packages/components/src folder, again using git mv and run npm init in this new package. Name the package as @my-org/components the best practice is to use npm scopes, that means we are prefixing the package with @my-orgBut it's your decision eventually.

Package dependencies

• Make sure we have all the dependencies of the component package added to this new package.json file.
• dependencies such as react, style-components, material-ui should be peerDependencies as the consumer app will install them
• Note: In case you have dependencies from another folder of your app, you will want to extract this folder to another package under /packages. Also, not always you want to export all of your components. Some might be very specific to your app and tightly coupled to the app logic. It's always possible to refactor and remove the coupling, but don’t bother doing this at the first step and keep the app's components.

webpack.base.config.js

At the root of the project, it’s advised to have a base webpackconfigurations that other packages/apps can extend. This file will contain our most basic webpack configuration. you can use webpack-merge to do a deep merge between the configurations.

Module references/package resolve

Once we are done with that, we need to fix all references to the components folder to use the new package instead. There are two ways of doing this.

We can support only:

import { MyComponent } from '@my-org/components'

That should be easy enough, but we will treat our package as a folder and for better tree shaking, although it’s not a must.

Instead, we can support:

import MyComponent from '@my-org/components/MyComponent'

That usually means that you have a folder MyComponent with index.ts file in it and the component's stories, tests, typing, and styling, yet it's not mandatory.

so we are going to do this in a few steps :

1. add @my-org/components as a dependency of my-app in its package.json
2. replace all references. You can do it easily with VSCode, hitting CTRL/CMD+SHIFT+H and using the regex: [\.\\]*/components to replace with @my-org/components
3. for eslint to resolve our package, we will add the following in webpack:

{
  ...
  resolve: {
    extensions: ['.tsx', '.ts', '.js'],
    alias: {
      '@my-org/components': path.resolve(__dirname, '../../packages/components/src'),
    },
  },
}

make sure that eslintuses webpackas the resolver, within eslint, "settings" you should have:

"import/resolver": {
      "webpack": {
        "config": "webpack.common.config.js"
      },

This should come along with eslint-import-resolver-webpack in your root package.json.

4. for typescript to resolve our packages, we should add in the root tsconfig file :

"paths": {
      "@my-org/components/*": ["./packages/components/src/*"],
      "@my-org/components": ["./packages/components/src"],
    },

5. do the same for jest using moduleNameMapper :

moduleNameMapper: {
    '^@my-org/components/(.*)': path.resolve(__dirname, './packages/components/$1'),
  },

Your IDE (I hope you are using VSCode) should already recognize the imported components, and by clicking them, you should be navigated to the correct component that exists locally.

Cross apps and packages configurations

Typescript

Note that we have now 2 tsconfig.json files. To have the same standards across apps and packages, keep the original tsconfig.json file in your root directory to apply across the whole project, and add a new file my-app/tsconfig.json that will extend the original one :

{
  "extends": "../../tsconfig",
  "include": ["src/**/*"]
}

Jest

Our jest.config.js will now be on the top level. for any special configuration, we can extend this file as well, so we can create another jest.config.js the file that will have :

const baseJest = require('../../jest.config');
module.exports = {
  ...baseJest,
  setupFiles: [
    'dotenv/config',
    '<rootDir>/jest.init.js',
    'jest-canvas-mock',
  ],
};

Another important configuration is pointing jest to the correct babel config, so in case you are using babel for code transpile, you probably want this as well :

transform: {
    '\\.js$': ['babel-jest', { configFile: '../../babel.config.js' }],
  },

Well, we did a lot of testing, and some of them involving other components. These new components are now in a different package, and therefore we might want to mock them. you can do it using __mocks__ the folder in your app package. See Jest manual mocks for more details.

This tutorial was written on the fly while actually doing these steps and changing a repo to be a Monorepo using npm7.

I hope you can find anything here helpful as this could be a painful process.