300 applications upgraded to Java 17 in one commit. The day before vacation. The power of monorepo!
It’s been six years since we decided to try a monorepo aproach for our microservices. Today we have over 300 applications in this monorepo. The applications are generated using templates. We refer to these applications as our “golden path” applications. All applications use Spring Boot and React and live in the same Git repository.
Friday afternoon, the day before Easter holiday, we upgraded all monorepo applications from Java 11 to Java 17. This was done in one commit and at rest. The next day I was off to the Canary Islands, and the rest of the team to the mountains. In this article I will share why this major change to more than 300 applications did not turn out to be a project in itself.
15000 tests to the rescue
But why not wait to after the holiday? The short answer is; why wait when you can do it today. All our applications have unit and integration tests. In the monorepo as a whole we have more than 15000 tests. When all of these run green, we are pretty sure that the change meets the required quality level. It is also important to note that we do not automatically deploy the applications to production. When we start doing this, it might be that we will not do such a change the day before vacation.
Sharing code is sharing knowledge. Local improvments in one team become global improvements for the whole company if this is done right. This is also one of the main benefits with using a monorepo.
One team to boost productivity
But let us travel to the Canary Islands. The island is lovely, especially because of the climate. The temperature in Easter is perfect for us Norwegians after a dark and long cold winter. One thing you dont’t want to do, is ruin the long awaited trip because of stress at work. Upgrading 300 applications from Java 11 to Java 17 can cause such stress. To optimize the developer efficiency at SpareBank 1, we have established a Developer Experience (DevEx) team. I am a part of this team, and we took on this upgrade task. The challenges with such an upgrade are much the same for all of our 300 applications. The DevEx team can gain expert knowledge when it comes to such an upgrade. An alternative approach would be to let every team do the upgrade themselves. That would certainly work too, but time spent on this task could have been used on building new features (opportunity cost). As teams have different goals, this upgrade might not be prioritized, and therefore it would have taken longer time to finish.
Handling different technologies have a cost. This cost is often invisible. Having a standardized way of making applications makes it easy to switch teams, share code and knowlegde, and to make tooling. Tooling includes local build scripts and shared build pipeline. Not having to handle different Java version is just one benefit of marching in step.
Dependencies can cause frustrations
Our applications share code at compile time. This code is what we refer to as our libraries. The library code is not versioned. All application is “on head” (using the latest library/shared code). This is an important principle for us, and a common technique when working with monorepos. This means that all changes in the library code must be compatible with all application in the monorepo. It is not allways obvious why this is a good thing. For a developer that wants his/her feature in production as fast as possible, it may seem daunting to have to change code in other teams’ applications. But one thing we have learned over the last 15 years is that this is for the best for SpareBank 1 as a whole. When using mulitrepo with versioned shared dependencies we ended up with what is often refered to as “version hell”. Application 1 depends on Library A. Library A depends on Library B. If you needed to do a change in Library B, and you needed to get it in to production fast, you had to build Library B. Then you would bump the version in Library B, build that library, and you probably understand, do the same thing for Library A. It did not stop there. All application that used Library A had to be bumped and built. But wait, some of the applications are on an old version of Library A. Nobody remembers how to refactor these appliactions to make them work with Library A. If that is not enough, you have to make sure your third party dependencies are in sync. What happens then? The world don’t stop spinning. The trip to the Canary Islands is tomorrow. And here you are, in a middle of a “dependency hell”.
In a monorepo a pull request will show all changes needed for your feature. Also if you need to do library changes in shared code. This makes it easier to do a review because you avoid the need to also review the library change in a separate repository.
Local discoveries are converted into global improvements
All code in the monorepo is easily accesible to all developers. They have quick access to the code from their IDE. The can copy it, be inspired or refactor it as library code. This is knowledge sharing. One good example is when one team switched from Webpack to Vite, it did not take many weeks before several other teams did the same. Another example is me coding a new feature. The probability is high that somebody else have coded something quite similar.
Deleting code increases your velocity
Developers are problem solvers and coding is one tool used to solve problems. Maintaing code has a cost. Often this cost is higher than writing the code. To save money we need to be able to delete code. In a monorepo where all code is on head, your IDE can show you all dependencies. Should you miss a dependency, the build will break for the applications affected by the change. Deleting code is also important, as reducing mass will improve the speed in the development teams. When doing multi-repo we rearly deleted code. We just deprecated it. Today we actually delete it. Over time this is likely to have a significant impact on the mass of code we need to take care of.
Peace of mind and happiness
So, how can we go on vacation with no stress when doing a big refactoring the same day the vacation starts? My answer is the confidence you get when all applications has been built and all tests have passed. Our pipeline also deploys all applications to our Kubernetes test-enviroment. This gives us confidence that the images has been built ok, and that the configuration seems ok. I say “seems ok” beacuse as we don’t automaticly deploy to production (yet) we can not be 100% sure. But sure enough that we can travel with our minds at the right place.
Want to try monorepo?
Monorepos are not for everyone. You need a mature development organization that is able to build the necessary tools and understand the value of a monorepo. It is not obvious to everyone that it is more efficient for the company as a whole if a platform team makes changes to other teams’ applications. Most people understand the value of having all applications always use the latest shared code, but not everyone sees why it often does not work to let teams themselves take care of this. Teams may have other priorities than updating to the latest version of the shared code. And the cognitive load that teams are exposed to, increases as they must understand all changes in infrastructure and shared code.
If you still want to try the monorepo way, I wish you good luck! Feel free to contact me if you need someone to talk to before or during your monorepo journey.
References
Velocity defeats itself. Get acceleration instead https://jessitron.com/2022/12/22/velocity-defeats-itself-get-acceleration-instead/
Monorepo med Git og Maven — hvordan lære gamle hunder nye triks (in Norwegian) https://2019.javazone.no/program/eb8bc683-ceaf-4baf-82ee-d7b72960a955
What is a Monorepo, Really? https://www.codesimplicity.com/post/what-is-a-monorepo-really/
