Modular Elixir at Fresha — Part 1: Introducing Scalpel
Development of a growing system in an efficient and maintainable way requires proper code modularization. Duh, right? There’s no single best way to do that though, no silver bullet — you’ll always want to adjust it to factors such as system growth, level of coupling or deployment requirements. And of course the nature and needs of the organization itself, such as time to market or independence of involved dev teams.
Elixir community proves all of these points by a lot of ongoing discussions around the purpose and usage of Phoenix contexts & umbrella projects or even the need for private modules. Discussions in which you may hear wildly differing opinions from some of the most respected community members such as Sas̄a Jurić, Dave Thomas, José Valim or Chris McCord. Discussions that are fascinating and eye-opening.
At Fresha, we’ve started to face a codebase large enough to demand a solution and patterns way above just a copy of Rails conventions that we’ve used to apply before we’ve embraced the Elixir/Phoenix stack. We’ve decided to come up with our own, opinionated approach to modularization of our system.
We’ll present our outcomes step by step in a series of articles called Modular Elixir at Fresha. In this first entry I’ll introduce our approach towards solving this problem. Fasten your seatbelts and experience our take on Elixir web app modularization. Ultimate, unprecedented, un… let’s just carry on.
A lot of what you’ll read in this piece may be reminiscent of early chapters from the excellent Adopting Elixir book. Indeed, this project is one of the cornerstones in our fascinating Elixir adoption journey.
The path to enlightenment
We call the entire effort Project Scalpel. That’s our fancy equivalent of Windows 95’s Project Chicago. But what took Microsoft two decades to come up with through Windows 10, we can humbly admit to just being born with — Scalpel is not a single close-ended release. It would be just as fun to revamp the entire project every time it evolves again as it was to reinstall good ol’ Windowses every time they went dead or needed an upgrade again.
Scalpel is rather a never-ending, incremental effort leaden and curated by our backend architects and technical leads to constantly adjust to the needs of our product. It’s supposed to yield a platform with optimal functional split and proper boundaries, but it’s also supposed to adjust to its growth and changes.
Although it may be a never-ending effort, we already have a solid foundation that we’re building on top of every day. And that we definitely consider worth sharing through these articles.
Initial assumptions
We build on and orbit around idiomatic Elixir and Phoenix, embracing its flexible & decoupled (at least compared to Rails) approach to code organization, but ruthlessly tailor it to our needs with a goal to mix pragmatism and quality for a set of conventions and rules that’s Just Good Enough™. For us, that is.
We intend to take a lot from Domain-Driven Design, but we don’t hesitate to steer away from it when we see it justified. And we surely don’t aim to be a group of fanatical followers on teachings of DDD. Which is accidentally kinda our only reasonable choice considering we’re still in shock & healing after finding out that Rails is not your application and as such we’re still wrapping our heads around the DDD principles.
Most importantly, we first and foremost focus on organization of the business logic layer. At least in the current stage of the project — although we do see a need to revisit our approach to the web layer and API design, we want to first tackle the most urgent and crucial regions. First things first.
Background constraints
First of all, we’re building a product that’s a single cohesive platform. Even though we’re currently functioning under two brands (Fresha for marketplace and Shedul for SaaS system) and even though they’re operated by different groups of users, they’re logically just two views on a shared set of core entities and logic around them. Therefore we see a great need for sharing the common code and accessing same data from many functional areas.
While the growing amount of that logic needs proper order and encapsulation, we mostly depend on database transactions for change consistency — even across multiple areas. It’s a great convenience and productivity booster. We do however want and need to use multiple databases for some special cases — isolated OLTP or OLAP. Oh, and our database is strongly established as a source of truth so… sorry event sourcing!
Our organization wouldn’t handle developing micro-services the way e.g. Netflix does it — i.e. by dedicating what from our perspective looks like a massive effort towards facilitating an infrastructure made of a variety of technologies and lots of independent development teams. At the current stage of company growth — where each feature’s time to market is so important — we cannot afford to sacrifice productivity. Yet at the same time we’re grown enough so that we must care about scalability — both in terms of feature delivery and platform performance.
Finally, we’re building on top of a legacy Rails monolith, but also on top of legacy Rails conventions that we’ve grown with as Rails developers and that didn’t simply fly out of our engineers’ brains due to switching to Phoenix (especially considering how familiar and cozy Phoenix may be for Rails devs). We have to take into account both the organization of existing code and the capacity of developers to adjust to the new.
All of these conclusions lean us towards a solution that mixes the best out of service-oriented approach and a plain good monolith. Something often called modular monolith. Something already recognized as awesome solution that allows to ensure loose coupling when properly applied.
Coincidentally, that’s exactly what Elixir and Phoenix seem to be forged for. But, again, they don’t provide us with one silver bullet template just for our system (no, Phoenix contexts ain’t such and nor are they supposed to be), so let’s next review what kind of dilemmas do we stand against given the specific assumptions and constraints that we’re facing.
Questions yelling for the answers
Up to this point, everything’s clear. But here’s where things get a little ugly…
- What are the distinct business areas in our platform that could be encapsulated in modules to hit the cohesion vs coupling sweet-spot?
- How do we split our code into business area modules with the ability to share common deps? With what means do we want to isolate them?
- Can business area modules be nested within other business area modules? What’s the meaning and consequences of such nesting?
- How do we want our business area public contract and private implementation code to be indicated, organized and documented?
- Who owns persistent data structures? How is that ownership indicated? What compromises (e.g. cross-area table joins) do we allow?
- How do we facilitate various types of interactions between areas in order to minimize coupling and enhance unit independence & testability?
- How do we ensure that all areas are clearly defined, evangelized to the entire team and evolved in a well-thought, controlled way?
- When and how do we deploy many runtimes assembled from subsets of our platform’s components e.g. to ensure resilience?
- How do we indicate, cope with, call and extend the existing legacy code that doesn’t agree with our endgame rules?
- How much of what we’ve agreed applies to our non-business code, e.g. technical entities like repos or interfaces like controllers?
- How are all these conventions & rules documented and how binding they are? How do we support and ensure that they’re respected?
…and more. Uhh, a lot we don’t know and a long way to go…
It’s a sublime mixture of project, domain, technical, infrastructure and organizational concerns. Some high-level and some getting into (perhaps seemingly unimportant) details. All standing between us and our perfect functional split, our “supercharged Phoenix contexts”.
But yeah — our goal will be to answer all of these and more! If you’re interested in how we wrapped our heads around these burning concerns, stay tuned for the next entry.
