Scaling our Ruby on Rails monolith using Packwerk (Part 1)

In this series of articles, we will share our experience with adding Packwerk to a large, fast-growing codebase.

The problem: is our codebase growing out of control?

Growing from 1 to more than 120 Ruby engineers working on our Ruby on Rails monolith in three years comes with its challenges. The codebase is enormous, with about 400 new commits to the main branch for a total of 45 000 new lines to our codebase every week.

Problem 1: The codebase is too large

The first problem that comes with this situation is rather obvious: reading and understanding 45 000 new lines of code per week is impossible, even for the most experienced engineers working at Pennylane since the beginning. The days when someone could know all the code by heart are long gone.

Worse, onboarding new engineers is getting more complex. We still have the same structure as a brand new Ruby on Rails application (except that we added app/services for our service layer), so opening config/routes, app/models, app/services, or app/controllers can feel overwhelming.

We need to find a way to make our codebase feel smaller for our engineers and especially newcomers.

Scaling our Ruby on Rails monolith using Packwerk

Problem 2: Dependency and privacy management

The second problem is related to code quality. In Ruby on Rails applications, all the constants are accessible everywhere, thanks to autoloading and convention over configuration. This is great because we don’t have to think about requiring them every time.

At scale, it’s also an easy way to shoot ourselves in the foot as it’s very easy to create tightly coupled applications. Whether in a model, a service, or a controller, you can take any ActiveRecord model, call create or update on it, and pass the attributes you want.

You can also call any service you wish to from any model or controller anywhere in the codebase. With everyone able to access everything from everywhere so easily, a Rails application can quickly turn into the infamous “Big Ball of Mud”.

The Big Ball of Mud

The main problem with the Big Ball of Mud is that it’s complicated to understand, especially in a vast and quickly changing codebase like ours:

• There is no privacy. All of our engineers can interact with all the constants in the codebase.
• There is no dependency management. All objects can depend on one another, sometimes resulting in circular dependencies.

All of this makes working on our application harder than it needs to be.

Now that our two problems are clearly identified, it’s time to create a group of motivated engineers to fix them.

Defining our ideal state

To solve the problems above, we first had a meeting to agree on how to improve our application. Here is the picture that we came up with:

The ideal state

Solution 1: Make the codebase feel smaller

We agreed that we first had to identify parts of the application that were almost independent and isolate them from the others.

Pennylane is an accounting software, but we offer additional services such as a professional bank account or banking integrations. All those broad areas of the product are mainly independent.

As engineers only work in one area at the time, by isolating those areas in the code represented by the black bubbles in the picture, we could make the codebase feel smaller and solve our first problem.

Solution 2: Dependency and privacy management

Even if those areas are isolated in the codebase, they still need to interact with one another. Those interactions should only happen thanks to an explicit and well-documented public API represented by the green dots in the picture.

We need to find a way to hide from the rest of the world objects not part of the public API to solve the privacy aspects of our second problem.

Last, we need a way to manage the dependencies between the different areas to avoid circular dependencies and make our code easier to maintain. This will solve the dependency aspects of our second problem.

Exploring different tools

Now that we have an ideal state in mind with ideas of solutions, it’s time to find the right tool to accomplish the job.

Micro-services

The first solution we talked about was micro-services. Breaking the app into micro-services would allow us to break the code into smaller parts. Doing so would help us untangle the dependencies. This would solve both the above-mentioned problems and help us move toward our ideal state.

We already have a few independent services at Pennylane, primarily for working with our data/machine learning teams for particular needs, such as providing matching suggestions between invoices and transactions.

However, moving from monolithic to micro-services architecture comes with many challenges and overheads. You need API calls for your services to communicate. You will also need a message broker for asynchronous communication. We wanted to avoid going down that road and solving our current problem by adding a new set of problems we would need to handle.

Packwerk

We follow closely the innovations that happen in the Rails ecosystem. We know that GitHub, Shopify, and Gusto are still monoliths, even though they are applications beyond our scale.

We especially heard about a tool from Shopify called Packwerk that they use to make their application more modular.

We decided to learn more about this tool. We discovered that combined with Gusto’s packs-rails library, it would allow us to split our large Rails application into smaller Rails applications easily.

All those Rails applications would live in the packs/ folder, and the packs-rails gem takes care of autoloading for us:

File structure of the banking and accounting packs

In the example above, the accounting and banking packs are two independent Rails applications living inside the packs/ folder at the root of our monolith. There are only two differences from a classic Rails application:

• There is a package.yml file to configure the pack;
• There is an app/public folder where the public API of a pack lives.

The package.yml contains three very interesting configuration options:

• enforce_privacy: true : This configuration option states that only constants that are defined inside the public API may be referenced by other packages;
• enforce_dependencies: true: This configuration option states that the current package should only depend on packages listed as dependencies;
• dependencies: This configuration option lists the actual allowed dependencies of the current package.

For example, the package.yml file of our banking package above could look like this:

enforce_dependencies: true
enforce_privacy: true
dependencies:
  - 'packs/accounting'

The banking package depends on the accounting package because when receiving transactions, we want to automatically create the correct accounting entries for accountants using our product to save some time. To do this, we will call a public method of the accounting package, therefore creating a dependency.

On the other hand, the package.yml file of our accounting package would look like this:

enforce_dependencies: true
enforce_privacy: true

We didn’t specify dependencies on the banking package here because accounting should know nothing about banking features such as synchronizing transactions, thus helping us avoid circular dependencies.

Once packages are properly configured, running the bin/packwerk update-todo will analyze the code and ensure the privacy and dependency configurations are respected. It will list all the violations in a package-todo.yml file so that engineers can later work on improving the code.

Thanks to Packwerk and the packs-rails library, we will be able to solve our two initial problems without adding the complexity that comes with micro-services:

• The codebase will feel smaller because the code will be split into smaller Rails applications, all living in the packs/ folder;
• We will be able to easily untangle our dependencies by configuring the dependencies of each package and properly enforcing privacy.

Conclusion

We already started extracting a few packs from our Rails monolith successfully. In the next blog post, we will share our extraction/migration guide and some code examples showing how we extracted routes, models, controllers, services, jobs, specs, and more in their own packages!