Coupling in Microservices, Part 2: Single vs. Multi-Repo
In my previous article, I discussed some of the advantages and disadvantages of single vs. multiple services. In this article, I’m going to assume that you went the multi-service route.
Some Terms To Think About
When we talk about “services”, there are actually multiple different things we can mean. (I know that these terms may not be the commonly used definitions!) In this article, I will be discussing three aspects of a service:
- A system is what I’ll be referring to a bounded context as. This can be a group of services working together to accomplish a common purpose, or a single service all on its lonesome. A Kubernetes pod can often be deployed as a system.
- A service — in the Docker / container world — is a single process. This can be an HTTP service, a scheduled job, a message consumer, etc.
- A repo is your actual source code.
In our own tooling at Flipp, as of this writing we assume that all three of these are closely connected. One repo provides one service which is responsible for one system. However, I’ve started understanding the limitations of this mental model and wanted to explore the ramifications of it. Most importantly — does it make sense to have one repo per service or one repo per system? I take it as obvious that you should not be sharing code across systems.
New KPI’s to discuss
In the previous article, I talked about KPI’s (Key Performance Indicators) and how I wanted to explore each KPI separately and see how one or multiple services affected each KPI in isolation. I’m going to do something similar now for one or multiple repos.
The same caveats apply as previously:
- Assume that the system uses some kind of relational database, although you can generally swap in a NoSQL or similar solution without much difference.
- Assume that all services are in the same programming language. There’s really no reason not to use multiple repos if you’ve got different languages.
(Note: one thing that can help to make multiple repos “more similar” to a single repo is tools like meta or git-subrepo, which combines multiple repositories into a “parent” repo.)
Speed of System Creation
How fast can the system be written from scratch?
The more repos being created means more time to complete the project — any new repo has a non-zero setup time. There are two aspects to this:
- Creating a new repo and populating it with a boilerplate or bare-bones system that has the correct access pattern (HTTP API, delayed job processor, message consumer, etc.) but does nothing;
- Shared business logic, schemas, etc. If you have more than one process that needs identical code, you’ll need to either a) copy/paste all the logic and make sure they stay in sync, or b) create a third repo to hold this logic and include it in the other two.
This seems to indicate that even if it’s almost trivial to set up a new service, it may be beneficial to have those different services running from the same repo.
Ease of Understanding
How quickly can someone understand your system — enough to answer questions about how it works?
This is possibly the KPI most open to interpretation and dependent on how good everything else is.
In general, having multiple repos makes it harder to understand how everything works together:
- Figuring out what uses a particular function or class means you also need to know where the access points are (HTTP, message consumers, etc.) and which other service inside the system (or outside it!) calls them;
- Tying together how a feature works across services means having to read/search through multiple repos and sets of code.
If you have external documentation, or a consistent pattern (all HTTP services look like this, all delayed jobs look like this), you can mitigate some of the pains of having multiple repos. In addition, understanding a smaller piece of the puzzle (how this process contributes to the feature) can be easier when the repos are separate.
Of course single repos can easily be more confusing if they aren’t well-written. If you have spaghetti code where one piece of code affects five others when they have no business being anywhere near each other, you’ll run into more problems than you’ll solve. Multiple repos is a way of enforcing encapsulation; you need more discipline to enforce it in a single repo.
Long story short, bad code is bad code no matter where it is, but introducing multiple repos also introduces some level of additional complexity even with good code.
Ease of Onboarding
How quickly can a new team member start being productive?
Other than ease of understanding (above), the main determinator is how easy it is to set up their environment so they can verify that what they’re doing actually works.
For simple tasks, it can be faster to become productive when repos are separate. You only need to clone the repo you need to fix, and you (e.g.) don’t need to set up a local message broker if you only need to add a feature to the UI. If this is the bread-and-butter of your workload, multiple repos can be a fine choice.
For more complex tasks, though, where multiple areas of the system are affected, multiple repos become more challenging. You will need to clone several repos, set up your environment several times (e.g. running npm install ), and figure out a way to get your services talking to each other in your local environment. You may need a tool like rvm to install multiple versions of the same language or library if your services use different ones.
Using docker and docker-compose can make this easier and less error-prone, but comes with its own extra set of challenges, especially for junior devs, in terms of the extra layer of execution and understanding how to interact with containers. It also can be slower to run (especially on Macs) and harder to connect debuggers from your IDE’s.
Feature or Bug Implementation
How quickly can a defined problem be solved?
If the task is simple and the service responsibility is well-defined, fixing it in a single one of multiple repos can be faster, since the repo itself is smaller and easier to reason about.
However, if the task is larger, you’ll need to make changes in multiple places, put in multiple pull requests (each of which may have its own back-and-forth code review process) and likely have to do some kind of end-to-end test. In addition, your repos may have different language or framework versions, meaning context switching when fixing one or the other .
In a single repo, you’ll never need more than one pull request, one ticket to manage the process, and can likely write a unit or system test inside the same repo.
Having multiple repos can enforce encapsulation and make each area simpler to change — but you can get the same benefit by using a single repo that has well-defined code boundaries.
One downside of having a single repo is that you risk having your change affect other areas of the code inadvertently. If your system repo is relatively small, this is less of an issue; but if you start building up many services, you’d need tight unit tests with broad coverage to ensure that new features don’t break existing code anywhere else.
Schema Migrations
How easy is it to make changes to how your data is structured?
If your system lives in a single repo, you only need to make the change in one place. Your unit tests will then automatically pass or fail and you can handle everything once.
If you have multiple repos, you’ll need to propagate the changes to each repo separately. Either you’ll need to copy/paste the changes, or you’ll have to have a shared library or sub-repo that has the changes, which will then need to be updated in all repos. This is not only more work that will take more time, but increases the chances that you’ll miss some service somewhere which will be using the wrong schema.
There are tools that can make this process easier, like the Confluent Schema Registry. However, all this does is make sure that downstream systems don’t break if the upstream schema changes. It doesn’t actually propagate the changes down to other systems transparently.
CI/CD Integration and Deploys
Here, multiple repos come out the winner in terms of creating tooling in CI to run tests, do deploys, etc. It’s very easy to separate out deploys of different services if each service is in its own repo. It’s still possible to do it with a single repo, but you either have to deploy everything at once, or get clever (separate by directories, have a mapping file, require manual action) with your deploy process.
Note that running unit tests could be wasteful if using separate repos when one can do. Much of the time in CI is spinning up images, running dependencies like databases, installing packages, etc. If you could do that once and run your tests for all services in the same workflow, you’d save a lot of processing time and likely cost.
On the flip side, in single repo-land you could instead lose more time by running all your unit tests when you’re only changing some of the code.
Summary
Each of these KPI’s have a lot of ifs ands and buts, which was largely my goal. As in all things software, there is rarely a clear winner for any given design. The answer to almost any architecture question is “it depends”.
Advantages of Multi-Repo:
- Enforces separation of concerns
- Easier to understand individual pieces of the puzzle
- Easier to make simple changes
- Less chance that making changes will affect other code
- Easier deploys in CI/CD
- Faster unit test runs in CI when making smaller changes
Advantages of Single Repo:
- Faster to create a new system
- Faster to understand the system as a whole
- Quicker onboarding / environment setup
- Faster to begin investigating problems (not necessarily finding the solution)
- Faster to make changes that affect multiple areas
- Easier to make changes to schemas or shared business logic
- Faster unit test runs in CI when making larger changes
In general
My suggestion would be to use one repo for a bounded context except in the following situations:
- Your Continuous Deployment pipeline can’t handle the single-repo-multi-service pattern and isn’t flexible enough to build it in;
- You don’t have broad coverage of unit tests in your code or the willingness to write them; or
- You have multiple teams working on a bounded context at once and need to split that context down as much as possible to reduce the blast radius.
As always, look at what’s important to you, what resources you have available and where you want to focus to decide what direction to go in.
Summary — My Values
I can’t speak for everyone at Flipp, but my own set of values could be described thusly:
- People cost more than machines. Reducing development time is one of the most useful things we can do.
- Retaining people is important, especially retaining good people in today’s engineering market, where the average time at a company is two years. Things we can do to improve morale give a large benefit that often outweighs the downsides.
- Being agile (in the sense of moving quickly) gives the team more freedom to create an architecture that actually maps to the problem they are trying to solve. If it takes a long time to spin up a new service, or there are technical limitations stopping that service from doing what it needs to do, we may find ourselves picking the “wrong” architecture to save time.
Based on this, my order of operations for all the KPI’s mentioned in the last two articles would look something like:
- Ease of understanding
- Ease of onboarding
- Speed of system creation
- Speed of feature/bugfix implementation
- Ease of investigations and debugging
- Ensure smooth schema migrations
- CI/CD integration
Based on this, my preferred (not only!) pattern would be to have a single-repo multi-service representation of a single bounded context system. This allows us to keep models and schemas shared, and have a single place to search, document and setup, but allow us to scale separately and reduce the blast radius. The downsides may be more difficult CI/CD integration and deploy logic, the possibility that a service may have access to resources it doesn’t need, and some lowered performance. I’m good with those given the advantages!
What’s your priority list?
I plan on also writing up some case studies about how we can use these KPI’s to drive design and architecture of a system — keep your eyes peeled!
