Building monolith as a learning ground for a well-designed microservice architecture
The idea with microservices is that instead of having one server running a gigantic codebase called a monolith, we break it down into multiple smaller services that communicate with each other. The advantages of microservices have been heavily advocated in the tech community and are solving problems from scalability, development speed, modularity, and robustness. But on the other hand, there are also horror stories of microservices architecture going wrong, where the engineering team had to spend months either rewriting it back into monolith because of the wrong decisions being made in the split.
Our strategy was to build a monolith system, to get useful learning about the domain and infrastructure. We will in this article investigate why that is a good idea, instead of going for a microservice architecture from day one.
Unclear domains
One of the main reasons for not going for microservice first is that the product has not matured enough, and the business domains are therefore unclear. One known example of what this could lead to is SnapCI which had an experienced team and started out with a full micro-service architecture with 9 micro-services, a sophisticated messaging queue, monitoring, and automated infrastructure and deployment. But what they experienced was that every time they were developing a new feature, they often had to change the data structure for multiple micro-services. This led to a high cost of feature development. After several months of fighting this, SnapCI ended up refactoring the micro-services back to a monolith.
Prematurely decomposing the system can be costly, and this is one of the reasons that it could be wise to start with a monolith and wait for the domains to settle before starting to decompose things into micro-services.
Over-coordinating services
One of the advantages of microservices and a more modular system is that each team can work more independently of each other, instead of creating pull requests that touch each other’s code which leads to complex merge conflicts. In other words, you might increase parallel work and team autonomy with a service-oriented architecture.
Technology like Kubernetes tends to make the teams create more granular services, as the initial cost of creating new services is small. But if the team is small, or you go too granular with the modules, the team will spend more time in building communication and coordination between the modules and the operation complexity of running them, rather than working on the business logic without the complexity. This also makes refactoring and API changes difficult, as you might end up coordinating and deploying to multiple services when releasing new features that quickly lead to bugs. You are now not dealing with a monolith, but a distributed monolith that is harder to work with. This leads to the same problem as with unclear domain — longer delivery time.
Microservice overhead
When you are starting with building microservice architecture, you might need to deal with things like eventual consistency, multiple databases (and potentially a data platform), tracking errors across multiple servers, distributed authorization, software updates on multiple servers, and messaging queues like Kafka or AWS SNS/SQS. Database transactions over multiple services are also not easy and need a workaround for example using Saga pattern or 2PC protocol. Also, testing and debugging becomes harder, as we are now searching for bugs across a distributed system instead of one server.
One of the bigger questions when starting to design microservices is also communication. Should you mostly on sync communication or async? If you go for mainly sync communication, you might get into latency problems and would need to battle that through investing in caching and a binary protocol. If you go for mostly async communication, you would get into the problem of eventual consistency and the additional overhead of duplicating data.
There are known solutions for all of the challenges, but distributed systems introduce complexity that you might not need to get out of a feature. By starting out with microservices, we will spend months building out the microservice architecture before you can even build a feature. We wanted to focus our effort on delivering customer value as fast as we could instead.
Technology fragmentation
The beauty of service-oriented architecture is that every microservice can use its own tech technology. One service could use Node.js, another one could Scala, and another one could even use Rust, and it will all work beautifully together. The argument is that not every problem is a nail, and every solution is a hammer, and we should rather use the right tool for the right job.
But just because you can, doesn’t necessarily mean that it is a good idea. First of all, letting every team decide their own technology, makes it harder to change teams or to have engineers across teams help each. You will then not just need to learn the new domain, but also potentially new programming languages and frameworks. Robustness will also be decreased, if the team suddenly stumbles on roadblocks with the framework, as the expertise is now spread thin over multiple technologies, rather than focusing on becoming experts in some selected fields. The last point is you will miss out on writing common libraries that could benefit multiple of the services, for example, logging and tracing library, event library, workflow engines, and more.
By starting off on a monolith and building up competence in this technology, and having everyone onboard on moving in the same direction, we could have knowledge and library sharing across the teams. We of course don’t need to start off with a monolith to have everyone aligned on the same technology, but having a monolith could help.
When is the right time to transition to service-oriented architecture?
As we grew our engineering and product team last year, we experienced the pain of having everyone working on one codebase, committing and deploying things to production every day. We came to a place where it was hard to track every change that happened and had multiple teams working on the same codebase that affected each other's work with no real ownership. Also, our monolith had been running for years and was huge with millions of lines of code, making it hard for newcomers to get a good overview of what was going on. Our databases were starting to have various performance issues, and by having all the domains using the same database, it was hard to isolate the effects. As our tests increased, our deployment of the monolith increased from 10 minutes to 1.5 hours(!!). In other words, the complexity of the monolith was becoming huge and slowed down productivity.
On the other hand, the domains had settled, and we were able together with the product organization to easily draw clear boundaries and create solid teams around the domains where we could slice the user journey into pieces that made sense and evolved around clear KPIs. This made it easier for us to create a clear plan of how to split our monolith into stable microservices with clear boundaries and contracts between them.
We came to a point where it made sense for us to invest our effort in a service-oriented architecture — even with the added “microservice cost”. As we were building out the monolith, we were hitting on some of the same challenges where the solutions needed the same infrastructure as micro service-oriented systems, for example messaging queue and authentication system. This made us able to build out parts of the microservice infrastructure and focus on one challenge at a time. Over months, our infrastructure was redesigned into a distributed system and could let the teams “peal” out microservices from the monolith, like layers on an onion - starting with the ones with the least dependencies. In the end, it’s all about tradeoffs, but we believe that the learnings we achieved with the monolith were extremely helpful for now building our microservice architecture.
