TL;DR
A system with no clear architecture is doomed to become a legacy system after enough changes in the requirements over time.
I’ll introduce the very insightful “IDesign method” and explain how it can help with creating a solid architecture that will withstand the test of time.
Motivation
I’m sure you’ve all had, at some point, a small cute software project. You were proud of it, enjoyed playing with it or even enjoyed just looking at it. But time passed, requirements changed and much like in the movie “Gremlins” your cute little project turned into a horrid monster. One so scary you can’t look at, not to mention fix its bugs or add new features.
There are many possible reasons for how that could have happened. Usually, it is either because there was no architecture (3-tier doesn’t count) or that there was architecture but it was not built for future changes.
So at this point, I hope you are convinced that we need an architecture that will withstand the test of time and will provide clear guidelines for all new features and fixes.
— OK, OK architecture is important. But who has time for this?🤨
In his book, Juval says that it might take weeks or months to find the core use cases and areas of volatilities but explains that that is requirement gathering and not design. Design should take a few days and over time a few hours.
On top of that, I think the following quote sums up the motivation nicely
“If you think good architecture is expensive, try bad architecture”
Brian Foote and Joseph Yoder
“The Method”
In Nielsen, we take pride in doing things right and following the common patterns and best practices. But unfortunately, when it comes to architecture, there is not much out there.
I don’t mean to say there is no information at all, many of you heard of DDD (Domain Driven Design) or “Clean architecture” for example but like most other material on architecture, they offer insight and paradigms but are but not a straightforward methodology to create architecture that is easy to teach and follow over time.
We ended up using a design method invented by Juval Löwy*, a Microsoft software legend, and master architect, to help create a common language for building and following software architecture.
In this post, I’ll shortly explain the key principles** and give a few examples.
*Juval Löwy founded the IDesign consulting firm. They also offer training courses which are the source of most of the content in this post
**IDesign method also talks about how to breakdown tasks and manage projects but I’ll focus only on the design part
Functional decomposition
To understand how you should build your architecture according to “the method” we’ll start with what you shouldn’t do.
When presented with requirements we should never (never) build a solution that matches them perfectly.
— Wait 🧐, but that’s what we all do no?
— It might be… But how is that working for you? 😉
Designing a solution according to requirements in the IDesign world is called “functional decomposition”.
If you get requirements A, B, C for an application, functional decomposition would be to create service A, service B, and service C. The easiest way to explain why is that wrong is that it is a fact that the requirement will change, they always do. And when they will, it will break the design and will lead to one or more of these problems:
- Client is doing orchestration — hard to test, hard to scale to different clients (e.g. mobile and web)
- Duplicating behaviors across services
- Explosion and bloating of services
- Services are “stitched up” together (because features are represented as services and not as integration of services)
- Couples multiple services to data contract
- Difficult to reuse the same behavior in another use case
- Couples services to current use cases and their order of execution
If you will only take one thing from this post it should be this — never design against requirements!
Volatility based decomposition
The dictionary definition of volatile is
liable to change rapidly and unpredictably, especially for the worse
— It sounds like every requirement spec I ever had! 😆
Similarly, an area of volatility in our system will be a required behavior that is likely to change and that if not encapsulated can “break the design”
The metaphor used by IDesign to picture encapsulating volatilities is imagining your design is a room full of fireproof vaults, a requirement change is a ticking time bomb that you will throw into the right vault, where it will explode without affecting anything else in the room.
— So how do we find these volatilities 🤔?
There is an entire process here but I’ll try to simplify it into 3 steps
- Extract core use cases — what our system should do (forever and ever)
- Find areas of volatilities — what can change over time (that we should prepare for)
- Define the services that will compose the architecture
Extracting core use cases
From the requirements and interviews with the relevant stakeholders, we will define use cases.
We should try not to focus on the functional requirements (“The system should do A”) as they are likely to cause ambiguity and confusion. Different stakeholders (and different developers) can have a different understanding of how the functional requirement should be addressed.
Instead, we should focus on how the system should behave i.e. what are the use cases of the system. A use case will be a set of activities to achieve some value for the business.
After we have all the use cases (that we could think of) we will group them into 3–5 core use cases — according to Juval every system will always have 3–5 core use cases, never more.
To validate, we would go through all of our use cases and make sure that they are all variations of the existing core use cases
Another way to think about the core use cases is with this thought experiment: If we were to write a brochure that explains what our system does, what will we write in 3–5 bullet points
Here are the core use cases we extracted for a recent actual project*
- Create content rules for a segment
- Manage my segment taxonomy
- Buy/Sell segments (marketplace)
And here are some individual use cases:
- Create a segment rule based on boolean logic of other rules
- Auction segments for sale with given starting price
- Duplicate segments (including rules)
- Disable a segment (including its children)
- Search segments according to partial segment name
*In our domain a “Segment” is a profile that describes online devices (e.g. Man between 20 and 30 years old), “rule” is the logic that our engine will run to decide what devices are part of the segment and a “Taxonomy” is a hierarchy tree of segments
Areas of volatilities
There are 2 axes of volatility:
- At one moment in time, how two different customers interact with the system? (e.g. power user and end consumer)
- Over time, how is one customer’s interaction with the system likely to change? (e.g. client requesting additional capabilities after working with the system)
With these possible change axes in mind, we will go over all the use cases and think about what could change.
Common volatilities could be:
- Volatility in how a user interacts with the system (e.g. “view only” vs. power user)
- Volatility in notification medium (e.g. e-mail, Slack)
- Volatility in payment method (e.g. debit card, Paypal)
- Volatility in data scale
Define the services
In the IDesign method there are exactly 5 types of services:
- Client — Handles communication with client, no business logic (e.g. REST controller) — who is making the request
- Manager — Orchestrated business use cases, define the workflow — what needs to be done
- Engine — Executes business logic — how to implement an activity
- Resource Access — encapsulates accessing resources (e.g. DB, REST endpoint) — where do I get data from
- Utility — Cross-cutting concern that is not specific to our business logic (could be used in a coffee machine)
There are a couple of different rule sets on how to communicate between these services. We went with the common approach of “Semi-open”:
- Flow control only goes from top to bottom
(client → Manager → Engine (optional)→Resource Access) - Each Service can access any service, as long as it’s top to bottom
- Manager can call other managers but only by triggering an action (asynchronous)
- Every service can access any utility service
- Each service should be independent (potentially a microservice), this means that for instance, it has to have it’s own business objects
Having only 5 types of services with a clear scope for each is very powerful. It creates a common language when talking about components and makes it very clear to understand responsibilities and boundaries.
The constraints on who can call whom and how helps with removing coupling and reducing complexity.
Now that we understand our building blocks and how we can put them together we can compose our architecture
The architect’s mission — to find the smallest set of services that will satisfy all core use cases and encapsulate all volatilities.
Since all use cases are variations of the core use cases, and since we encapsulated every volatility by a single service, it means our architecture satisfies existing requirements and should satisfy future changes without breaking.
Get down to business
Once you have the services laid out in the architecture you can start planning the design and implementation.
Our approach to implementation was to first decide on the API of every service. Now we could, one feature at the time, focus only on the relevant parts of the API of each service and implement them.
That way we got to a working MVP pretty quick and were also able to test it very extensively — end-to-end tests, integration tests, and even stress tests.
Here are some additional tips for the design and implementation of the services:
- Managers should be as declarative as possible and mostly contain a set of use cases to execute on each flow
- Only use engines if they are needed (i.e. there is volatility in how to accomplish the specific stateless activity)
- Write declarative integration tests that describe business use cases and use as little mocking of components as possible
- Client services should not contain business logic and should never call more than one manager
- Make sure that your utilities can indeed work in a coffee machine (no context of your domain)
- Arrange your project according to the core use cases (e.g. under core use case folder we will have client, manager, engine and resource access folder), utilities will be on the same level as core use cases
- choose the right message bus technology for you (we use RabbitMQ) and make sure it is well maintained
Summary
I think we can all agree that architecture is important. Building software without it will surely turn your cute baby project into a 3 headed beast.
With the right methodology, designing architecture should be easier and easier to create over time, and more importantly easy to follow as the requirements change (and they will).
IDesign offers a very nice solution for such methodology but the important thing is to have something structured that everyone you work with can understand.
I’ll finish with a very nice quote from Juval Löwy
For the beginner Architect, there are all sort of ways to do pretty much everything, but for the master there are all but a few
Additional information
To get more information about IDesign you can check their website or read Juval’s new book that explains in detail everything I mentioned and more.
You can also check out these youtube links:
