tsrc — Handling multiple git repositories without losing your HEAD
Git has become the mainstream source code manager and is the de facto standard for many projects, both open and closed source.
However, it does not scale well with huge code bases commonly found in the enterprise world.
Some companies decide to put everything in one giant repository. By doing so, they often end up patching existing source control software as Facebook did with Mercurial.
Another way to do things is to keep using good old git: by splitting the sources across multiple repositories.
Managing complexity
In the “multi-repo” case, how do you keep track of all the repositories, and how do you synchronize changes among them?
One popular method is to elect a ‘master’ repository and manage the others ones through the use of git submodules. Unfortunately, it is not always possible to implement this master/slaves design. Furthermore, the ergonomics of git submodule
are not always on par.
tsrc
Enter tsrc. We use it every day at Tanker to manage our many sources:
At its core lies the concept of a `manifest`. It’s a simple YAML file that lists repositories by their relative paths in the workspace (`src`) and by URL:
repos:
— src: foo
url: git@gitlab.local:acme/foo
- src: bar
url: git@gitlab.local:acme/bar
That means all repositories are listed in one place.
If you have multiple teams, you may want to use groups so that each team can select which subset of the repositories they need. Each group has a name and a list of repositories:
groups:
first_group:
repos: [a, b, c]
second_group:
repos: [d, e]
This approach is known to work with at least 300 different git repositories :)
Usage
To use tsrc, all you have to do is put the manifest itself in a git repository, and then use the following
commands to create a new workspace:
$ mkdir ~/work
$ cd work
$ tsrc init git@gitlab.local:acme/manifest.git
You can select one or several groups with the -g/--groups
option.
Making sure all the repositories are up to date
Once the workspace is created, you can update all the repositories at once by using tsrc sync
.
- The manifest itself is updated first.
- If a new repository has been added to the manifest, it is cloned.
- Lastly, the other repositories are updated.
Note that tsrc sync
only updates the repositories if the changes are trivial. tsrc will not touch any repository if:
- The repository is dirty
- The local branch has diverged from upstream
- There is no remote tracking branch
It is up to you to solve these issues any way you see fit. This way, there is no risk of data loss or sudden conflicts to appear. tsrc focuses on doing simple tasks right and never tries to do smart things.
Lastly, so that you know where manual intervention is required, tsrc sync
also displays a summary of errors at the end:
$ tsrc sync
…
* (2/2) spam/eggs
=> Fetching origin
=> Updating branch
fatal: Not possible to fast-forward, aborting.
Error: Failed to synchronize workspace
* spam/eggs: updating branch failed
Managing merge requests
As a bonus, tsrc is also able to automate review workflows, whether you are using GitHub or GitLab.
More info about all of this in the documentation.
Conclusion
We hope you’ll give tsrc a try and that you’ll find it a handy tool for your own projects.
tsrc development happens on GitHub.
Feel free to contribute, open issues, and/or give us feedback in the comment section.
PS: tsrc is made with love by the same talented people working at tanker.io. We provide an open-source privacy solution that’s as user-friendly as possible. Sign up and take it for a spin here: tanker.io