Mono Repository or Poly Repo? We go Hybrid!

Photo by Noah Buscher on Unsplash

The number of repositories is a long time dilemma for both startups and big companies alike. In Outbrain (>100 developers) we started off with a mono repository code-base on a Subversion server. A couple of years ago, we started to move to Git and along with a Micro-Service architecture, decided to split our codebase into multiple repositories.

Now we are evaluating a new approach.

When we worked with a mono-repo there were a couple of big advantages.

Developers could easily reuse code, all of the code-base was built for any service, so any Maven module was available to all services.

Refactoring was relatively easy. All the code was checked out in one IDE project. When a developer wants to do a method rename, for example, it was as simple as every IDE refactor command.

Also, looking for code examples was simple.

All in all, it made communication between developers and API’s effective and agile.

But… It became slow… and IDE became sluggish….

We started seeing developers arriving in the morning and updating their trunk, compiling it (from the command line of course) and going to grab a coffee until it is done.

In addition, we didn’t have enough barriers and started to get All-Engineering spammy emails ‘Sorry, I broke the build, Fix is on The way’ on a daily basis.

Then we moved to multi-repos

Generally, we allowed a team to take out their libs into external repo’s while still having the monolith trunk depend on modules from it by publishing those modules artifacts to Artifactory.

That allowed teams to be more agile. Build time was shortened (for teams going out of the trunk) and it was easier to deploy code to production. In addition, it was easier to update dependencies since not all our codebase had to be aligned anymore.

On the other hand, after many teams moved out, we started to notice dependency conflicts AKA jar hell. Conflicting versions of 3rd party libraries- as well as our own- were finding their way into production in the form of ‘NoMethodError’ and ‘ClassNotFoundError’ et al. Our internal libraries were permissive in the number of dependencies they brought in, and also backward compatibility guarantees and documentation were missing in action.

Where are we heading now?

Before I will describe what we will try to do now, stop for a second. If you did not understand the problem or the situation check this post that has a great explanation about the trade-offs:

We are now trying to move into a Hybrid-Repo. We will have one repo for JVM infrastructure libs and cross-teams API and communication, while services and utilities will continue to live in separate repositories, one per team.

Let me emphasize and rephrase — with Hybrid-Repo we will have one repo that is responsible to keep shared libs and API between teams. We will strive to keep it fast compiling and lean by practically avoiding slow tools like Scala and slow or flaky tests like tests that access Database.

Other languages like javascript Node.js and python are out of the game since in this post I am talking only on our backend services written in JVM languages.

What do we expect to get from our hybrid approach?

Easier communication between teams — similar to the way we get with mono-repo, changes will be transparent and easier to coordinate.

Reduce dependency conflicts — since we have one place with a repo that manages those dependencies. Now repositories with services will have to upgrade only one dependency instead of a couple of versions from multiple teams. It means easier versions alignment.

Have a repo setting the bar for code quality and norms with high standards.

Keep a high level of agility and freedom of teams inside their services repositories that are still split per team.

What are the risks?

Get back to slow-compiling-heavy-repo — It might be that as the organization gets bigger, there is a size that this repository will not fit anymore. First, we try to mitigate that by hierarchy inside the repository itself, means every team will have its own directory of modules and can usually work on that directory only. Second, we will try to enforce standards. For example, we will prevent slow running tests from getting in or slow compilation (Scala…).

When it grows more than we can work with, I hope we can split it to more than one isolated responsibility if we will need to, but we are aware this approach might get to its limit as well.

More thoughts on the subject

Mono repo vs Multi (or Poly) repo is a flame war these days but is actually an old question with many answers.

Essentially I see it as a subjective choice that an organization makes which is related to tools and culture.

There are tools that make you work with one and feel like the other. Git submodules allow looking at multiple repositories as one. Meta repository also allows something similar. There is also a Repo tool for Android that can be used with other languages as well.

Sourcegraph (which is now open source) will allow searching multiple repositories as if they were one.

Upsource as many other code review tools will allow making code-reviews in a centralized location.

The Bottom line: I think that as for today there is no one answer that fits all, and the solution should be tailored for the specific requirements and needs of the organization.

--

--

--

Outbrain is the world’s leading native advertising platform, guiding the digital discoveries of consumers around the globe. Genuinely connecting marketers, publishers, and the consumers in-between, Outbrain serves more than 308 billion recommendations, organically personalizing,

Recommended from Medium

So You Want To Create a Database With SQL

Implement a doubly linked list with Python

The 2 Qualities to Look for in a Partner………

6 Exceptional GitHub Repos for All Developers — Part 1

Praise A Barber And Warn People About A Barber

Complete Guide to Terraform: News, Updates, Tips

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
osha1

osha1

More from Medium

Monorepo Setting Using YarnBerry & YarnWorkspace

9 Most Useful Storybook Addons and Features Through Examples

PDKit: Project as Code

Managing source code with a monorepo