The Monolith That Breaks Up Itself

Marianne Bellotti
Aug 26 · 7 min read

How small engineering teams can build distributive systems

Image for post
Image for post
Image by freepik

I actually really love monoliths. But I’m the sort of engineer that wants my tool box to be full of tools, even weird looking ones that only become useful in very specific circumstances. The advice I typically give people about maintaining technology is to embrace the inevitability of Conway’s Law and factor in team structure, composition and size into architectural decisions. Small teams tend to build monoliths because small teams tend to be monoliths.

Monoliths also offer slightly better performance than service based architectures. At least at first. It’s only once products reach economies of scale that the benefits of being able to build, deploy and scale different components of a system separately outweigh the increased latency of making a network call when you could use a library. Of course when it’s time to make that switch the organization has reached a level of growth where its primary focus is competing, competing and more competing. Breaking up the monolith while also rolling out new features is like remodeling your house while having a dinner party at the same time. For a small company, new features and new sales will almost always win out. Once a small company has become a mature company the task of breaking up the monolith has been spread out over multiple teams and increased in complexity.

At Rebellion we had another reason to sometimes favor more monolithic approaches. Government networks can be complex and unfriendly to cloud. That can mean operating on a private, possibly “air gapped”, network with edge devices where we can reproduce a cloud like experience without connecting to a traditional cloud environment. An architecture that can be coupled or decoupled easily can offer advantages.

All Rebellion products are built on top of a framework called servicecore. At the heart of servicecore is a gRPC server, then built into servicecore are many of the tools engineers need to build good applications anyway. When engineers use servicecore they get monitoring for free, they get tracing for free, they get proper logging and health checks for free, ORM integrations and easier migrations too. They also get things that can be enabled as needed but are turned off by default, like an interface for Rebellion customers to manage access control roles in a self- service fashion. But each one of these options is itself built using the servicecore framework, meaning they can be run as separate services or combined into a tightly coupled monolithic application with no real code changes.

Bricks and Studs: Code like Legos

Much has been written about designing applications in order to facilitate microservices. For example, domain orientated architecture organizes like functionality together so that larger services can eventually be split up. But most advice in this space doesn’t help you reconfigure code easily. It’s hard to break services off and even harder to pull them back together if for some reason the division ended up being a mistake. You have to design solutions around places where functions can logically connect in order to have that level of flexibility.

When building a monolith that can break itself up the first question is how should two sets of functionality be coupled? The most basic way is for the code to exist in the same place, and then when the application has scaled to the size where it makes sense to maintain two different services the code must be moved into separate repositories and rebuilt to be run independently from one another.

Another solution is for one service to be implemented as a package and imported by a host service. This keeps the code for each domain separate and means multiple services can reuse the same parts, but often requires the host service to build out endpoints and interfaces that interact with the package logic. As soon as we need a database, or a frontend, importing a package isn’t so convenient.

But by building the framework that allows those elements to either be run on their own server, or injected into another server, we can couple and uncouple sets of functions as needed.

One service injected via the servicecore framework into the gRPC Server of another

At the heart of servicecore is an application object (well… this being Go and all it is in fact a struct, but object is a more familiar term to most programmers). An engineer looking to build a service on top of servicecore need only specify the configuration they want in the application object and pass that into servicecore. Servicecore iterates through all the application objects, gathers and sorts the configuration, creates the gRPC server and passes the full configuration on to that server.

Image for post
Image for post
Servicecore parses application object + plus built in applications for logging, auth, etc into a gRPC server

For even the most basic service, there are multiple application objects, because the tooling that makes servicecore valuable to developers are all themselves application objects as well. Therefore, servicecore makes it possible to couple services together in two different ways: either we can use the application object to attach the endpoints and all their logic and dependencies to the same server, effectively extending it. Or the application objects can inject middleware to change the functionality of the server’s endpoints.

Middleware Is Back in Style

Admittedly, I cut my teeth as a software engineer during a period where middleware was the villain of architectural patterns. Middleware was “software glue” that in the wrong hands could change the behavior of a function in a way that was hidden from the software developer debugging it. But modern day web services have a lot of overlapping needs. Behaviors like logging and authentication have to be executed for every single request, every single time. Who wants to throw a good tool out of the toolbox just because other engineers might misuse it? I remember working with frameworks where to avoid the middleware curse the architect wanted you to use boilerplate code calling those functions in a transparent way at the top of each endpoint. Leave off the line that checks if the user is logged in? Guess that page is now public.

The opportunities to add middleware fit into two categories. They either execute before the function (pre-hooks) or after the function (post-hooks). We can determine some of the options a framework like servicecore could expose by tracing the path of a typical request through the application. No matter what the particular endpoint is supposed to do, a request moves through the following stages:

Stages of a request: transmission, authorization, execution, and response.

Every endpoint must receive a request, confirm that what is sending the request has permission to do so, execute the function tied to the endpoint (potentially triggering queries to other services or databases) and deliver a response.

Since we’re using gRPC we have a built in way of hooking into the transmission and response stages via interceptors. Even if we’re streaming data, we can attach logic that is executed before the request data is passed to the actual endpoint. This turns out to be a good way to implement that authorization phase. It can also be used to add logging for easy debugging.

Then there are hooks related to the lifecycle of the service itself. We definitely want to allow configuration to be passed between the host service and the extension. We’re using Cobra to do command line interfaces for our services, so servicecore can include hooks to add commands either when Cobra is initialized or at runtime for the server.

These connection points give us a lot of flexibility to add functionality to products without scaling up the overhead on the engineering team by asking them to both maintain the code and an ever growing fleet of servers in clusters per service. The complexity of distributed systems is ultimately valuable, but only once the usage levels on the products themselves have scaled up enough to justify it. Before then running services decoupled is just another set of responsibilities added to the work load of a small team.

Great Power and Great Responsibility

One of the things that servicecore does is enforce conventions. By creating a framework that packages up configuration and creates a gRPC server for you, we can add validation logic that ensures how things are named and some aspects of how they behave are consistent.

But the other side of building a framework like this is that it could be used to produce bloated services that are difficult to understand and debug. One big issue is the order in which applications are loaded into servicecore. Duplicates are ignored and hooks are executed in order. Changing that order might impact outcomes in ways not obvious to all engineers. The potential problems of middleware have not gone away.

Servicecore is called servicecore because it’s meant to serve as the core of services, not full products, but it’s also easier to maintain products when you’re able to give ownership of a set of functions to a 1 or 2 person team and not burden them with separate monitoring, on-call rotations, and deploys.

We’re a small team now, but we’re growing fast. Check out our open positions.

Rebellion Defense

Rebellion builds products using artificial intelligence and…

Marianne Bellotti

Written by

Lover of legacy systems and complex system failure. Opinions my own.

Rebellion Defense

Rebellion builds products using artificial intelligence and machine learning that serve the mission of national defense for the United States, United Kingdom, and our allies.

Marianne Bellotti

Written by

Lover of legacy systems and complex system failure. Opinions my own.

Rebellion Defense

Rebellion builds products using artificial intelligence and machine learning that serve the mission of national defense for the United States, United Kingdom, and our allies.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store