Monorepo Migration Guide
Backgrounds
Concept introduction:
Project Structure
The structure of the project before migrate:
The structure of the project after migrate:
Requirements:
- Keep git log
- Keep branches and tags
That’s the background, Let’s start the migration!
New structure
The simplest Mono repo structure is to just put different projects into different subfolders. Create three subfolders in the new project: backend, frontend, and mobile, which correspond to the original projects.
After migration, every time you need to create a branch or tag, you only need to operate one project, which is more convenient than the previous operation that creates branch/tag with the same name across multiple projects.
But we can do better!
Now that we have merged into one project, we have the ability to modify the code in bulk. For example, batch version number, automatic create versions and tags. There are a few tools that can help us do this, such as lerna.js:
The project structure using lerna.js looks like this:
├── lerna.json
├── package.json
├── packages
│ ├── backend
│ ├── frontend
│ └── mobile
The contents of the lerna.json file:
{
"packages": [
"packages/*"
],
"version": "1.1.2"
}It internally records a version field that represents the version number of the entire project.
At the same time, each item in the packages, namely backend, frontend, and mobile, has its own package.json file, which stores its own version number.
If there is a bug in the backend, we fixed it and decided to release the new version, you can first submit the bugfix change, then run lerna version patch to update the version number. Lerna.js will update the version in the lerna.json file to 1.1.3, and also detect the package (ie backend) that has changed between the last release and the current release, and set the version in the package.json of the corresponding package to the latest version number. Finally, the version in the two files lerna.json and packages/backend/package.json becomes 1.1.3, and the rest remains unchanged.
For more information about lerna.js, see its official website:
Migration tool
There are a variety of project migration tools that can implement the Multi to Mono transformation. I will introduce the import subcommand of lerna.js and the tomono tool.
In simple terms, the import function of lerna.js is simpler, suitable for simple projects (only the master branch), and the tomono tool is suitable for complex projects. Here are some comparisons for reference. If you are not interested, you can skip to the next section.
As can be seen from the table above, the advantages of tomono are obvious. I used the tomono tool during the migration process. You can get this tool in Github, which is a shell script:
Tomono tool needs to have a repo list file
// note：giturl name path
git@github.com:<yourname>/backend.git backend packages/backend
git@github.com:<yourname>/frontend.git frontend packages/frontend
git@github.com:<yourname>/mobile.git mobile packages/mobileExecute the following command to start the migration:
cat repos.txt | ~/tomono/tomono.shPreparation before migration
The fewer branches, the better!
The first thing to do is to merge the branches that can be merged and resolve the conflict. Avoid resolving conflicts after migration.
In addition, the tomono tool will iterate through all the branches and tags in the given repo sources, and then create the corresponding branches and tags in the new project. But one thing to note is that not all branches and tags in the new project include backend, frontend, and mobile.
If the structure before migration is:
Note that the branches owned by the backend, frontend, and mobile are not exactly the same. Both backend and frontend have a branch A, and both backend and mobile have a branch B. All three projects have a master branch.
For this case, after the migration, the structure is as follows:
Note that after the migration, branch A contains only backend and frontend, while B branch contains only backend and mobile. This is because only part of the original project has A and B branches.
To put it simply, which projects have a branch before the merge, and which projects are available in this branch after the migration.
Therefore, the following prerequisites must be guaranteed before migration to avoid future conflicts.
- All branches to be retained must exist in all subprojects. If it does not exist, you need to create it based on master.
- All branches to be retained must be matching the code in all projects.
- Merges all branches that can be merged and resolve conflicts.
- There can’t be conflicting branches or tag names, such as bugfix and bugfix/123, which conflict with each other.
Migration process
Migrate with tomono
$ cat repos.txt
git@github.com:myproject/backend.git backend packages/backend
git@github.com:myproject/frontend.git frontend packages/frontend
git@github.com:myproject/mobile.git mobile packages/mobile$ cat repos.txt | ~/tomono/tomono.sh# This step creates a folder called core and has the following
# structure
# ├── packages
# │ ├── backend
# │ ├── frontend
# │ └── mobile
In some cases, the core directory is empty after the tomono is executed, which is usually caused by an error in the process. You can check the error message given in tomono to troubleshoot the problem. A relatively common problem is caused by branch and label name conflicts, such as a branch named bugfix and another called bugfix/123, which conflict. To solve this problem, delete the relevant branches and tags, and then do the migration again.
Merge all branches that can be merged
Because the migration process will have git rewrite, it will bring some strange conflicts. Before continuing the operation, you should perform the merge operation and merge the mergeable branches once.
Push to the remote repository
As of now, all the changes are still local. After the above steps, we now have a folder called core, which has the newly imported packages directory and .git directory.
To push to a remote repository, you first need to create a new repository in the remote management page. After creating an empty warehouse, Github/Gitlab will give you follow-up instructions like this:
git remote add origin <remote repository url>
git push origin master
Also, refer to the Github guide:
There are all branches and tags in all related subprojects in the core directory. You don’t need to delete them. You only need to push the required branches and tags to the remote repository, others are ignored automatically.
After all the required branches and tags are pushed to the remote repository, you can move the Core directory to another location as a backup and then re-clone the remote repository to get a clean MonoRepo.
If some branches are the same as the master, such as a development branch that has just been merged into the master, you should not push it to the remote repository, but you should re-create the same name branch from the master (to avoid possible conflicts).
Post-migration matters
Init lerna.js
Lerna.js is not required, but it is very helpful, especially for JS projects.
lerna initThe detailed user guide can refer to the official website of lerna.js https://lerna.js.org/
CI/CD
If you are using CI/CD, then Mono repo can be a problem.
If you are using a newer version of Gitlab CI, it provides the only changes / except changes feature that can be configured to trigger the corresponding CI Task only if a given file changes.
Other CI/CDs will probably have similar features.
Let’s take Gitlab as an example to illustrate the CI/CD settings.
- Merge CI/CD tasks and create separate aliases for them. For example, in the backend, frontend, and mobile projects, the original CI has a deploy operation, which needs to be renamed to deploy_backend, deploy_frontend, deploy_mobile, or other names that can be distinguished.
- Modify the task trigger condition and add the only changes condition on the original basis:
In this way, you are done.
Reference reading
Mono repo tool collection:
