Going from a monolith to a monorepo
A step-by-step guide
Most of the publicly available documentation talks about moving from a multi repo to a monorepo. But what if , like us , you had to move from a monolith to a monorepo? In this article, I’ll share our experiences, all the steps of moving away from a monolith to a monorepo, the challenges we faced, how we tackled them, and the impact on the team.
First of all, let’s review our stack. At AB Tasty, we use:
- Webpack as our bundle tool
- Babel for exporting ES5 code
- Typescript to type all our projects
- React and Redux
Optin for a monolith, a good idea… or not
Historically, our main application, called “Dashboard”, was working along with two subsidiary projects:
- “API client”: a client exposing simple functions to communicate (read/write) with our backend APIs
- “Component Library”: a library of React components reused across our application, directly implemented based on our design system (buttons, titles, forms, …)
The Dashboard would call API client and Component library to function correctly. But the fact that they were three distinct projects made it quite difficult to maintain them. Whenever a new feature was developed, the three projects needed to be modified accordingly. This inevitably led to conflicts among the projects or mismatch of versions. As the projects were still at their beginning, and we were making numerous breaking changes.
A year ago, we came up with the suggestion to gather all three projects into one single repository. This meant a unique build, and one single deployment process and simplifying the versioning of the projects. However, we soon realized it had one major issue. From now on, the three projects were closed in on themselves, which meant that only the Dashboard could use the module’s API client and Component library, and our external teams — including our innovation team — needed to use them too. We tried to export the libraries, but they lost their typing during the operation and were unusable. Also, some links between the projects started to appear, coupling them. The projects were becoming unsustainable. We realized we just made a monolith.
That’s when we started thinking of a monorepo.
From a monolith to a monolithic repository: a 6-step guide
The idea was to go back to three independent projects but to keep hosting them in the same repository in order to have a common development experience. This way, we could benefit from the monolith’s main advantage: any cross project change can be done and reviewed as a single “atomic commit” and we can deploy all projects in a single command. But at the same time, the three projects could be exported and used independently in other projects.
Step 1 — Add lerna as a dependency of the project 🐉
Lerna is a library that helps you manage your monorepo. It uses a Lerna.json file that describes the configuration of your projects, with all the packages they are composed of and their dependencies.
So, I added Lerna as a dependency of the project, hence creating the lerna.json file that I modified to match our configuration. Fortunately, as the projects had been merged not so long ago, the links between them were not yet very strong, which let me disentangle them more easily.
$ yarn add lerna
$ yarn lerna initOnce the lerna.json configuration file is generated you can configure it with your project needs:
{
"packages": ["modules/*"], // The location of our modules
"npmClient": "yarn", // Either yarn or npm
"version": "1.1.0-beta.0" // The version of the modules
}I then added a new package.json in each of the modules to describe them: name, version, scripts, and dependencies.
Step 2 — Define our versioning format ⏳
In a monorepo, all the modules can use independent versioning: for instance, our API client V2 can be used in our Dashboard V1.3, but we decided to keep the same version for all the modules in order to avoid awkward problems, even if a project hasn’t changed. This forced us to create extra versions, but it helps us know in a glance which versions are compatible with one another.
This meant that I had to adapt the continuous integration (CI) scripts. In our prod and pre-prod environments, automatic jobs check the code, test it, and build it whenever a change has been made on Gitlab. Normally, Lerna only publishes new versions of the packages that have been modified. We had to add a — force-publish flag in the command line in order to bypass the check of the new code and force the publication of the new version of all the packages at the same time.
$ yarn lerna publish patch — message "[skip ci] Production release %s" — git-remote upstream — force-publish — yesStep 3 — Add the scripts and configuration to each module 📦
In a monolith, only one configuration or one set of package.json scripts is used. We had to separate it for the three projects. Fortunately, the old configuration of the different modules was still accessible.
Module by module, starting with the smallest and easiest API client (which only has few dependencies with external projects), I started mapping the current process, our package.json scripts, in order to replace them with a process using Lerna, without affecting the development experience.
To do this, I started adding the configuration and scripts for each project:
- Build
- Test
- Format — format the code with prettier
- Lint — to spot code or syntax errors
- Types — to export typescripts in order to improve the use of projects
For instance, when we launch Lerna in the root project, the Build command will automatically call all the Build commands as defined in each project of the monorepo.
In our root package.json:
"build": "lerna run — stream build",In other projects:
// The main application package.json
...
"build": "webpack -p — env.production",
...
// Api client package.json
...
"build": "tsc — project tsconfig.json",
...
// Components library package.json
...
"build": "webpack -p — env.production",
...Step 4 — Fix the bugs and miscellaneous dysfunctions 🙈
Modifying the imports
Lerna manages the way the projects interact with each other as npm dependencies and thus through node_modules. With the new monorepo, the Dashboard calls API client and Component library as dependencies. As such, the import statements must point to a node_module instead of the file directly.
Changing the aliases
In the past, we were using aliases to import modules from API client and Component library into the Dashboard, to avoid using relative paths.
> For instance, instead of writing “import Something from “../../api-client” we were using an alias: “import Something from “@api-client” (that we declare in typescript configuration). With the new Lerna configuration, I decided to keep the aliases to avoid changing too many files at once and increase the chances of losing myself in the modifications. So, I only modified the aliases to point to the node_modules of the projects instead of the folders.
"paths": {
"@api-client": ["../../node_modules/@abtasty-org/api-client"],
"@api-client/*": ["../../node_modules/@abtasty-org/api-client/*"],
}Making the projects work independently first
One of the advantages of the monorepo is the possibility to merge common configurations and dependencies, for instance the jest config to run tests among the projects (or ts config or even webpack config). But I chose to keep them independent at first in order to make sure they worked well individually. I’ll leave the merging part for a refactoring afterwards.
Refactoring exports
Previously, some files were directly imported with their relative path into the Dashboard. In the new architecture we can only import from the entry points of the other projects, it occurred that some specific functions/components were not exported outside of their resulting module bundle. So, I had to refactor exports to make sure everything is accessible outside of the modules, improving the structure of our code at the same time.
Step 5 — Document the change and communicate 🕵️
Once the three modules were up and running in our brand new monorepo, I adapted the README file and embedded development documentation to reflect the change in terms of commands and processes for the developers of the modules.
I also took the time to explain this architecture change the front-end developers at AB tasty, for everyone to understand the reasons, the modifications, but also to ask their feedback. Reactions were enthusiastic! The front-end devs appreciated the fact that it wouldn’t change their development experience, as all the commands remained identical. This architecture change is far from being a revolution, but it really improves the reusability of our modules and the overall experience of the developers around these three projects, by providing a stricter architecture.
Step 6 — Deploy 🚀
We communicated about the monorepo beforehand in order to avoid friction and resistance, and to answer all questions beforehand. But we had to wait for an “opening” in the deployment schedule. It’s now done: all the front-end developers switched to the monorepo weeks ago.
Unsurprisingly we encountered a few errors due to the developing environments (Windows vs. Mac) and some hitches caused by another project, calling the monorepo as a dependency. I was expecting such “hazards” but they were quite easy to fix. I simply updated the documentation along the way.
From a monolith to a monorepo: lessons learned
And there were quite a few:
- A better structured code
One of the unfavorable effects of the monolith we didn’t anticipate was the entangling of the projects. By doing a code review of the projects, I discovered some dependencies among the projects that shouldn’t exist.,I had to add some missing exports to make sure everything was accessible from outside the modules,I realized that using this kind of architecture will force us to have a better structured code.
- Communication is key
As we are working with multiple feature teams, this type of project can impact quite a few people. Communicating about the project early on, with all the developers that will be impacted, but also with the management, can help it go smoothly. Your manager should be able to support you by freeing up time to work on it and prioritizing it when other teams such as Quality Assurance are involved.
- Don’t wait too long to push to prod
If you embark on such a project, you have to realize that it is quite difficult to maintain and to anticipate all potential issues. It’s better to get the proper time allocated to such a project and to prioritize it to push it to prod, as it will impact other projects too.
- Constraint by design is better than a code review
The monolith allowed dependencies among the project that shouldn’t be. For instance, there is no technical constraint of using Dashboard in the Component library, but in terms of architecture it’s nonsensical. Instead of clearing the code afterwards through a code review, using a monorepo instead of a monolith enforce proper links between modules. It makes us think through when an error occurs and improve our code right from the start.
Impacts for the future
Going from a monolith to a monolithic repository helped us get a better usage of our core libraries. Our projects are also better structured (with a separation of concerns), helping us maintain a readable and scalable code base. Finally, it has opened the door to include our legacy project (coded in angular) in the same repository, easing the development process. Going forward, we will continue to gradually replace the features that need to be improved.
Have you made such a big change inside your company? Please share with us how it went in the comments section!
