Break Your App into Composable Modules

Over the years, my company’s main application has been under continuous development, evolving and gaining many features. Naturally, as time went by, our engineering team developed many frameworks and utilities that were needed for the app. For example — performance tracking, background worker system, analytics, storage access and even our own ORM, and much more.

This application is designed as a monolith. All of those frameworks are built on the same codebase with the app, and in many cases they are even coupled to the application’s domain-specific code.

As our engineering team (and company as a whole) matured, we started to get requirements for new applications — it could be an in-house dashboards app or managing deployments system, but also real new products that required actual production systems to support them. We started designing those systems, and guess what — we found out that we needed storage, analytics, performance tracking and all that stuff here as well…

This is where things get interesting.

What is the biggest benefit of a system that has been around for a few years? It has seen production. It met real users. With traffic. It was under load. It went through a lot of optimizations and improvements over the years.

Over time, systems as a whole become more robust, and as do their frameworks.

At this point we started to think differently. We can no longer write frameworks that are bound to a certain application. In order to really scale things out, we need to build reusable modules and libraries, which will provide the building blocks for all our applications.

We’ve changed our mind-set to module thinking

The plan was to break our main app’s frameworks into separate and composable modules that have great API.

The process in high level is:

• Identify and map. It turns out it’s not always simple to think about which frameworks you have. You may have the obvious ones like I mentioned earlier. But you may also discover along that way that you once wrote that cool utility class for handling REST API call to 3rd parties, which is also usable for the next application you are building.
• Extract. Write each framework as a separate project with its own API. The idea is that the next application we are going to build can be easily based on a selected group of components (you’re not always going to need them all). A recommended way to do this, is to first extract it to a separate package/assembly/jar in the same project, but this time without direct dependency by the application. This way you can stay in context while refactoring, and at the same time decouple things relatively quickly.
• Use. After extracting, try integrating it back to the application - this time as a 3rd party library. You will be amazed how bad your API is when you are the user :)
• Fix your API. Improve the API until you’re satisfied.
• Document. Now that you have an awesome API, go ahead and add a README.md file to the repository, so that the module will be easy to integrate and use in future applications.

Technology and development process

Each module is written in its own project, and has its own tests, with its own CI process. The project is visible to everyone in R&D in a public repo in our internal Bitbucket. We use a package manager in order to manage versioning and dependencies between modules. Each push to master automatically releases a new version of the module to the package manager, and each consumer can decide whether and when he wants to upgrade.

The key here is an easy and frictionless development process.

If it’s not easy, it won’t happen.

From this point, it will be very easy to push changes and upgrade the modules.

The benefits of composable modules architecture

Reduced development time

Reduced creation time of new applications by creating a unified toolbox, from which the modules can be combined with the requirements of each app.

Reduced maintenance by writing once and using everywhere, instead of copy/pasting around. The bug fixes and upgrades are applied in one place.

Reduced boilerplate by creating better and understandable API. Prior to extraction, some of the modules were coupled to the specific app domain. Extracting the module, forced us to think about improving the API usage to be more reusable, and to uncouple the module from its domain.

R&D growth and organization memory

Many of these modules were written a long time ago, by people who already left the company. Addressing those modules forces us to get to know them and have a deep understanding of them. This enables us to master our tools (which is something I believe in), and take them for granted, and this will help us to grow as a team.

Increased quality of frameworks and apps

Since each module exists and is maintained in one place, and widely used across different apps, they automatically become more mature and stable, since they have more clients and receive more feedback. They also have their own tests and CI, which is a major win.

Some ‘do’s and ‘don’ts’ we’ve learned along the way:

• Have tests and CI for all modules. Obviously. No exceptions.
• Don’t extract a module for the sake of extraction. ROI. Extract only if there’s another consumer for this module.
• Don’t let the same engineer extract all components. This won’t scale, we want to share as much knowledge as we can. Haven’t you heard about the bus factor?
• Each component should be handled by more than one engineer. Again, the bus factor.
• Don’t create deep dependencies between your modules. Create a flat hierarchy of modules in order to avoid dependency hell. Here’s a great post about this subject.

What’s next for us?

In the future, we would like to have a true owners/contributors model. In this model, each module owner/s will be responsible for accepting PRs for his module, and will be responsible for all aspects of the projects such as creating and leading the vision and giving an introduction about it for new developers.

A contributor can actually be anyone from the R&D team. This will be a true open source mindset. This model can bring a lot of benefits, not only technical, but this post is getting to long so maybe next time ;)

Better visibility

We need to find a way to expose a “catalog” of the different kind of modules. This will create a very dev-friendly ecosystem for our project. Imagine a web-based portal that the engineers in your team can just browse through, see what possibilities are there, who the owner of each module is, and which applications are using a certain module.

“You said open source — why not do the real deal?”

Maybe we will… Stay tuned :)

Closing

Breaking our app into composable modules helped us in many aspects. We reduced development time by introducing a way to select the building blocks of an app and start rolling, reducing boilerplate, and focus our time and effort on domain logic and bringing business value. We also got to know our frameworks better by going over the codebase and getting our hands dirty. We made them better by creating great APIs.

The beautiful thing is that new frameworks are now written like this from Day One :)

Cheers.

Originally published at code-lodge.com on October 25, 2017.