The current state of Gusto’s modularity tooling

In my last post, I looked at how we are steering Gusto’s packaging and modularization work using a broadly applicable way to think about package structures. In this post, I aim to connect this to our tooling setup today. The previous article may not be the most riveting one — nonetheless, you should go back and read it — the most salient points for today are:

• Applications and libraries can depend on and include libraries but not applications. Applications interact with other applications via their APIs.
• The possible APIs for applications and libraries are very different, with the ones for applications being more restrictive.

How we got to today

We can roughly split Gusto’s journey of using packwerk for modularization into three phases.

• Adoption: A few weeks after Shopify released packwerk in the fall of 2020, we made our first packages at Gusto. After about six months, we had moved most of our biggest codebase into packages — slightly less than 200.
• Expansion: While most of the original packages weren’t very clean and had many violations, teams across the engineering org created more packages. Even if imperfect, packages showed value as a means of visually separating domains, showing the desired structure, and gradually moving towards it. We got to over 400 packages.
• Refinement: Having seen the trend of the ever-growing number of packages recreating the problem of the codebase being hard to approach, we knew we had to find a way to change the evolution. So, the phase we are in today aims to resolve the conflict between the sentiment “there are too many packages for me to understand what is going on” and teams creating new packages all the time. The insight discussed in the previous post led us to ask: “What are the applications here?”

Packages as applications?

When we asked which applications we thought were present in this codebase, we came up with about 20. 20! Way better than 400. If we could represent these, we’d go from 400 to 20. That would be a massive improvement in the system structure’s approachability. At this number, you can also map applications to products and features that our customers buy, use, and can distinguish.

The next question was: How do we go from 400 packages to 20? We could not do that. The teams appreciated that they could structure the internals of their domain using packs and did not want to regress from that. Instead, we created a few pieces of tooling that comprise our solution today: layers, product services, and nested packs.

Layers and product services

First, we tried associating all our packages with one of the target applications. It turned out that was not possible. Many packs didn’t belong to only one application. There were a couple of groups that stood out:

• The application harness (the Rails app root)
• Dev tooling that is needed for data and environment setup in non-production environments
• Rails base classes
• Utilities that provide functionality to many or all product services

There is directionality to how these groups interact. If we put applications (I will call them product services from now on since that is what we ended up calling them) smack dab in the middle of these four groups, we can think of these groups as layers. The rationale for these layers is that every time you go from a higher to a lower layer, the lower layer should not need to know anything about what’s above: The Rails app configures the runtime of the application but shouldn’t depend on anything, dev tooling needs to have access to models in the domain (product services), but the reverse is not the case, product services rely on rails base classes and utilities, but those should be devoid of knowledge about the product services’ domains.

A visual to the rescue! The diagram below shows all the layers. The highest layer, the application harness is shown as the outside box. All other layers are boxes inside. The green arrows indicate the order of the layer hierarchy. The relative size of the layer boxes is a nod to the expectable different relative sizes.

We refined our package structure by creating folders for each layer and moving packs into them. In addition, we used the packwerk-extensions layer protection to enforce the order of these layers through packwerk. Moving some packages into the non-product services layers means only packages belonging to product services are in the middle layer. The code in this layer amounts to roughly 80% of the codebase and a similar percentage of all packages. So, for this layer, we nested packs into product services.

The structure today looks something like this:

/packs
  /tooling
  /product_services
  /rails_shims
  /utilities

  # Some layers omitted for clarity. The application harness is not visible
  # here because it is only the root package.

Nesting packs

To allow teams to continue modularizing the internals of their domain, we nested packs into product services. That created the 20ish applications/product services as folders within the product services folder while making space for all the packages belonging to this product service. An immediate benefit of this change was the improved ability to see individual domains in focus.

At this point, the new structure exposed a new challenge: What is an API? Once again, I refer to my previous post: The differences between possible APIs when comparing applications and libraries are substantial. And if product services are our applications, what are all the packs within these product services?

Here is where I empathize with the problems Shopify engineering had with privacy enforcement, as discussed by Gannon McGibbon in A Packwerk Retrospective. His analysis culminates in the observation: “The utopia we had imagined simply did not exist, and the tool we thought would get us there was leading us astray.”

We know we want product services to be applications, so we know what restrictions we can place on their APIs. But what about the packs within them? In general, packs with product services act like libraries, so we know we can’t place any (generic) restrictions on their APIs. This observation may seem like detail at first glance, but it is the crux of it all: We knew that we should place restrictions on the possible APIs for some packs, but we had no way of effectively deciding for any given pack whether it should have restrictions and if so, which ones.

As such, the best we could do was share general ideas. One such idea that I am guilty of having mentioned many times and that causes quite a bit of damage is “You should probably avoid ActiveRecord at the boundary.” Its danger is that it is good for certain packs and extremely bad for many others.

The danger of “no ActiveRecord at the boundary.”

For an application, “no ActiveRecord (AR) at the boundary” is a perfectly sensible thing to demand. In fact, it is so deeply ingrained in how we build things that the suggestion of doing something else might seem absurd: Rails controllers return HTML or JSON — because they return data across the application boundary and over the network to a client. If you wanted to send ActiveRecord objects, you would have to marshal them, but even then, you would be unable to use the server’s application context. However, we are not bound by such constraints within one application. We can do things differently.

If we forget that we have more options for in-app API design, the results for system structure and performance can be disastrous. Take the resolvers backing a GraphQL schema within one domain as an example. When the different packs in a product service construct a graph tree structure, how would it be built if we observed the idea of “no AR at the boundary?” We would have something like a repository pattern that would internally call into AR models of a pack to convert them into data-only structures and return those. While we may want this in other cases, here, it can be said to add two layers of indirection. The resolver would parallelize or chain these calls depending on the exact request. Any repository we build will either be less potent than what we get from AR functionality, or it will reimplement all its complexity. In the latter case… let’s face it is not going to happen. In the former case, you can expect to see all sorts of loading inefficiencies: n+1s, eager loading, lazy loading, over loading, etc. If, instead, the resolver could communicate directly to the pack’s internals and use all the querying powers of AR, we would need to make no such tradeoffs.

To implement these insights, at Gusto, we decided to explicitly separate the API of a product service from the APIs of packs within a product service. Today, we are working towards a special product service pack, “_api”, which holds the external API for a product service. For all other packs, we work to enforce folder privacy (see below), which causes any usage from outside to cause what we call a product service boundary violation. We allow packages inside of a product service to use any protections they wish, but we restrict privacy and dependency enforcement’s scope to the other packs within the same product service to reduce violation feedback noise: Within product services, it is privacy and dependency as desired by the owning teams, between product services all teams are held to product service privacy.

The specific tool changes are

• Folder visibility packwerk-extension: This allows us to control access to product service internals for most packs while excluding specially denoted API packs that contain the API to be used when accessing a product service from the outside.
• Ignore patterns for privacy and dependency: These patterns allow teams to continue to use these enforcements, focusing on improving the internal structure of the product services they are responsible for.

The aspirational structure of a product service looks like this:

packs/
  product_services/
    payroll/
      _api #product service external API
      calculate_payroll/ #internal
      filings/ #internal
      # ...

Summary

At Gusto, we have iterated on our modularity tooling setup multiple times to make more and better refactoring tools available to our engineering organization. The latest iteration creates a new balance between the complexity of the internal structure and its relevance for our business domains. We believe that it will drive better modularization over time and allow us to add an important piece to drive the next step in our software serving our customers: the creation and elevation of well-designed and meaningful internal APIs that can become the precursor to the next product feature and integration opportunity into the Gusto ecosystem of product.

PS: Packwerk vs pks (packs)

One last update on tooling. We have effectively stopped using packwerk in favor of pks, the rust implementation originally authored by Alex Evanzcuk (formerly on our team!). It is compatible with the core features of packwerk. However, some of the features discussed in this post have yet to be backported to packwerk (e.g., generic ignore globs). Pks is superior to packwerk because it is much faster and can be used to parse all Ruby files for their constants instead of extrapolating them from Zeitwerk-loaded names, which is what packwerk does. This means you can use it to get feedback on even more constants in a Ruby application, not just those Zeitwerk knows about.

PPS: Join discussions on modularity in Ruby and Rails applications on the Ruby/Rails Modularity Slack server — you’ll meet quite a few Gusties there! Check out my other writing on modularity at my personal website, https://stephanhagemann.com, where you can also find my book, Gradual Modularization for Ruby and Rails.