Developer-friendly multirepo with mr-developer

Rules and tools to manage dependencies without pain when monorepo is not an option.

I am sharing here the good practices and tooling we set up in the Onna’s frontend team to manage our frontend applications and their dependencies.

Monorepo, sorry but no

It seems like modern frontend development has no alternative than mono-repository strategy to manage sources.

When I heard about this concept, my first feeling was it was a bad idea. I mean, putting all the code produced by 20 developers in a single Git repository?… well, what about directly moving back to SVN, right?

I guess I understand why monorepo makes thing easier when you are building a framework.

But when you are developing an enterprise project, my experience showed me it was a very painful approach.

I will not get in too many details, but let’s mention few points:

• one of the main idea behind monorepo is to unify the version management, which is indeed a very good thing for a framework; but in a company you sometimes maintain different products based on different versions of your libraries; there is no easy way to do that with monorepo, while it is super easy with multirepo;
• the CI can’t take it: if any given pull request about the API layer triggers all the end-to-end tests of the UI components, it is clear your Jenkins will melt down eventually;
• it is not always possible to draw a clear line between your development and its external dependencies, for instance at Onna, where I work, we maintain few opensource libraries that we use in our company products; obviously, the opensource libraries are in public repositories, while our products are managed in private ones;
• more generally, you do not necessarily want all the teams to have the same access rights on all the parts of the code;
• monorepo implies to set up a super smart building system, and I really enjoy simple and basic solutions, like if I need to create an app, I just do ng new myapp and then ng build whenever I need to build it (I mention ng here because we use Angular, but the same would be true with any other CLI).

Naive mutilrepo is painful too

At Onna, we are developing and maintaining 3 pretty big Angular apps. At the beginning, all the common code shared between the 3 apps was managed in a single package.

As this package was growing fast, we decided to split it: one package was dedicated to backend API calls, one for models and data logic, one for custom UI components, etc.

This isolation process was very positive, it made testing easier, refactoring easier, external dependencies upgrade easier. Basically, everything was easier (I guess that’s no coincidence if so many developers in so many technos chose multirepo and used it for years).

Everything but dependencies management.

It seems like nothing works fine on this matter, like if frontend development practices are not there yet, not mature enough.

Npm link?

npm link is the built-in solution offered by npm to use a given folder as a npm package we can install in any npm project.

It can be handy in some simple cases, but when working on a project, it just does not work. Let’s mention the two main problems:

• the link is global (it is shared in our entire system), and for obvious reasons, we do not want that, we want to work at the project level, for instance we want to be able to switch from our library current version to a working branch directly from the project;
• it is based on the library’s package.json and that’s annoying, because our package.json might not be accurate in dev mode, for instance, it will probably contains things like:

"main": "dist/index.js"

But our sources are not in dist, there are more probably in a location like src/lib for example.

Of course, we can try to cheat by putting a dev-dedicated package.json in src/lib so we can make the npm link at the source level directly, but the situation will turn out to be messy sooner or later.

Yarn workspaces?

yarn workspaces mitigate some of the npm link drawbacks, but that’s still not ideal.

Each workspace has its own node_modules, each of them need to be re-build locally if we want the changes to be reflected at the project level.

Rules for a good developer experience

My main angle is to improve the developer experience, and after trying different setups and solutions, I have identified 3 important rules to make sure our dependencies management will insure a pleasant and efficient experience.

1. All sources must be in ./src

In dev mode, your favorite build utility (Angular CLI or Webpack) watches changes in ./src.

When we are searching for a given button label or any piece of markup and having no idea where it comes from, it is much simpler to search it into ./src rather than our gigantic ./node_modules.

Let’s keep things simple and put all our on-going work here, including our work-in-progress dependencies.

2. Versioning must be based on Git

All work-in-progress dependencies must be Git checkouts, any mechanism based on package.json version number will be painful.

With Git, all use cases are simple.

For instance, our project’s 2.x version depends on our library 3.x version, let’s put it in 3.x branch. Whenever we push a change on 3.x, a simple fetch will retrieve it in our project 2.x version.

If our project’s 1.3.1 version depends on our library 2.8.19 version, let’s create a 2.8.19 tag, and we are done.

No big news here, I am just saying Git is good at managing source code versions, and NPM is not (NPM is a package manager, it is good at managing packages when their development is finished).

3. Unique build chain, unique ./node_modules

Our project dependencies are just sources, there is no reason to build them separately from the rest of the project.

Having separated build chains for dependencies makes the main project build chain more complex, and ruins our CPU (and our CI CPU too).

The same goes with external packages. Our project’s dependencies’ dependencies should just be our project’s dependencies, and they should be in ./node_modules.

That’s a job for Mister Developer

I come from Python, and more specifically from Plone, where we are used to work with tons of dependencies.

And in Plone, when we have to work on a project containing many packages of which we only want to develop some, there is a tool for that, named mr.developer.

A good idea is a good idea, whatever the technology. What we need is just a NodeJS mr-developer.

mr-developer console output

Basic usage

It can be installed as any NPM utility:

$ npm install -g mr-developer

And then it requires a file named mr.developer.json like this:

{
    "lib1": {
        "path": "/src",
        "url": "https://github.com/mycompany/lib1.git"
    },
    "lib2": {
        "path": "/src/lib",
        "url": "https://github.com/mycompany/lib2.git"
    }
}

And by running:

$ mrdeveloper

we get lib1 and lib2 repository checked out in our project in src/develop.

But as the actual sources are not necessarily in the root of each of those 2 repositories, the path attribute allows to indicate where they are located, and our project tsconfig.json is updated accordingly:

{
...
    "compilerOptions":
        "paths": {
            "lib1": ["develop/lib1/src"],
            "lib2": ["develop/lib1/src/lib"],
        }
...
}

By default, mr-developer will check out the master branches of each dependency.

But we can define some specific branches if needed.

Imagine we are implementing a new feature in our project which impacts both lib1 and lib2. Our current changes are managed in thefeature-A branch of our project, where mr.developer.json contains:

{
    "lib1": {
        "path": "/src",
        "url": "https://github.com/mycompany/lib1.git",
        "branch": "feature-A"
    },
    "lib2": {
        "path": "/src/lib",
        "url": "https://github.com/mycompany/lib2.git",
        "branch": "fix-something"
    }
}

Whenever we want to work on this feature, we just do:

$ git checkout feature-A
$ mrdeveloper

And our 2 dependencies are positioned on the proper branches.

We can also use tags, which are safer than branches to define a precise versions set (in production for instance).

Let’s our prod branch contains the following mr.developer.json:

{
    "lib1": {
        "path": "/src",
        "url": "https://github.com/mycompany/lib1.git",
        "tag": "1.3.6"
    },
    "lib2": {
        "path": "/src/lib",
        "url": "https://github.com/mycompany/lib2.git",
        "tag": "2.3.0"
    }
}

Once again, with:

$ git checkout prod
$ mrdeveloper

we switch from our feature A dev setup to prod setup, and we can then check a bug reported on production.

Also very handy for our CI

It is always very painful when a CI system uses NPM packages to retrieve our work-in-progress dependencies.

Any change in one of those libraries will require the CI to run a first time to build the library package, then we can update the project package.json to upgrade the library version number, and then we can run the CI again to build the project.

That’s endless.

mr-developer makes it much more pleasant.

Let’s say we pushed a change in the fix-something branch in lib2, we just need to launch a new build on the project’s feature-A branch, the CI will run mrdeveloper so it gets our last lib2 changes integrated in the project build directly.

If you don’t like TypeScript, that’s fine

The previous examples are related to Angular projects.

Obviously, it works exactly the same for any TypeScript project (even if not Angular), as it only relies on the usage of tsconfig.json.

Nevertheless, mr-developer is also able to write libraries paths in another file:

$ mrdeveloper --config=jsconfig.json

jsconfig.json is a file very similar to its TypeScript equivalent, but most part of time is only used by the IDE and not for the actual build.

But a small hack allows to read its paths to inject them as aliases in webpack.config.js:

const pathsConfig = require('./jsconfig').compilerOptions.paths;
const alias = {};
Object.keys(pathsConfig).forEach(package => {
    alias[package] = pathsConfig[package][0];
});
...
resolve: {
...
    alias: alias
}

Disclaimer: non-TypeScript use cases haven’t been tested in production context for now, we are Angular people, we don’t use React, but feedbacks are very welcome!

Update 2020–09–24

Two important updates:

Firstly, mr-developer is now obsolete and has been brilliantly replaced by mrs-developer (and related command is now missdev)

Secondly, I have changed my mind about monorepo, tools like Nx make it very comfortable.

Still, as we are still actively developing opensource libraries while we are using them in our apps, we do need a way to checkout external Git repositories in our Nx project.

To do so, mrs-developer is perfect. The following command will put our opensource dependencies in the Nx’s/libs folder.

missdev --output=../libs

Vice-versa, if we have to use a monorepo-based dependency, mrs-developer supports multiple package declaration for a single repository:

{
    "angular": {
       "url": "https://github.com/angular/angular.git",
       "packages": {
          "@angular/core": "/packages/core",
          "@angular/forms": "/packages/forms"
        }
    }
}