Modular Architectures: setting up for success
Success of the modularization exercise is determined by not what you are about to do, but the context in which it will be done and what you already have.
This article is Part II in the series on “Modular Architectures”. Part I can be found here.
When you go through the exercise of large-scale restructuring — refactoring is a subset of restructuring — you have to make choices. You have to limit your scope to only those areas that you give the biggest benefits and you will to do it in multiple cycles.
But to be successful and effective, I realized that it heavily depends on the context under which it is performed and what has already been done.
This article is an acknowledgement of the support I received and the set up that was already there before the refactoring. I merely exercised the “adjacent possible”. These are the edges that I extended further.
On the technology side
Modularity Patterns
I knew that the code was hacked many times to provide quick wins for the business. Such quick hacks, referred to it by some as incurring “Technical Debts”, leads to fragile codebase, higher defects and longer release cycles. Technical Debt is about borrowing time from future for delivering quick fixes and the interest payment of this debt comes in form of recurrent cost such as the ones listed. When you gain enough of such debts, it cripples the business and demoralizes your development team. Trust me, you don’t want to be there.
Apart from quick wins, business context and many original assumptions have changed as well. It was about time to adjust the design to suit the new realities as well.
I knew from the very beginning that the design had to be restructured to make the application simple, flexible and easily extendable. Yes, it does sounds like cliché but the intent was honest. How do I adjust the design in a way that addresses current problems but has necessary infrastructure to curb or at least inhibit from acquiring large Technical Debt in the future. But at the same time the design has to remain and simple and flexible to support evolution.
My research for the solutions led me to a book by Kirkk Koerschild called “Java Application Architecture: Modularity Patterns With Examples Using OSGi”. Although, even after reading the book I was not convinced to use OSGi, the book provided me with the right set of framework, not a software one but a mental framework, and the confidence to pilot the design thinking in a small portion of the system. This is what paved the way for full restructuring.
It made me realize that in large application, physical design is as important — if not more — as logical design. Physical design is what effectively delivers you the trifecta of simple, effective and extensible code.
Separate User Interaction Module
There is this perpetual fight to categorize a flaw as a design flaw or as an architectural. If you attend a “root cause analysis” meeting for an issue, you will invariably hear that the “architecture” is flawed from non-technical people. This irritates me. But I also understand that it’s not their fault. For them, design and architecture are one and the same. I distinguish them by considering cost, time and risk involved to fix it. When the flaw is based on an early decision, it will be expensive and risker to fix and is, therefore architectural, otherwise its a design flaw.
One of many decisions while creating an application is to choose the framework for User Interactions. This is an architectural decision. If we made a wrong choice, the remedy is going to be expensive. Not to mention some heads rolling.
Business constrains and user expectations made us to decide to not to use a typical MVC layer model for the application. Instead we decided to create a completely separate UI module that would only rely on data using APIs from the server, delivered over HTTP. We settled on using GWT as the client side technology.
It was how application was built, before the refactoring exercise. This architectural decision has exceeded in delivering the experience that our users wanted. It served us well during the restructuring as well.
If you have an application where user interface is tightly integrated with server side code, like Spring MVC or Ruby-on-Rails, I have to be honest; it can make your restructuring exercise exponentially difficult and risky. Yes, I can understand that you can create modular solutions using any framework. But in my experience, most of them focus on logical modules and not physical.
Think “API First”.
Separate Domain Model
Domain Driven Design emphasizes on three things —
- Focus on the core domain.
- Create models in collaboration with domain practitioners and software practitioners.
- Speak “Ubiquitous Language” with explicitly bounded context.
We decided early on to create model of the domain that can match the business as close as possible. If you have gone through this exercise you will understand that this is a process to find the balance between the “idealist” architect, “here’s how I want” enthusiast business representative and a “realist” developer. We went through multiple iterations before we could settle down on a stable domain model.
Again, this is done before refactoring.
Our fanatical pursuit of creating separate domain model, devoid of any presentation, business rules or persistence logic, was a big help in applying modularity principles later on.
External System Integrations — Context, Boundary and Adapters
Our application had integrations with three external systems. We had to use different technologies for each of these integrations. One was done through screen scrapping; another was using their internal API and a third one with “REST” like Web API.
All of these integrations had very clear business context, boundary and thus resulted in us creating separate adapters for each of them. These adapters were then wrapped through an “anti corruption” layer. Our system was communicating with external systems only through these anti corruption layers.
This separation helped us in further refactoring them as physical modules and enabled us to focus on modularizing internal ones.
Groovy as a Rules and Workflow Engine
This one should be tagged as “Essential Complexity versus Accidental Complexity”.
As in most projects, we have complex business rules and need some kind of workflow tool that can handle business process orchestrations.
Selecting on off-the-shelf Rules Engine and an ESB would have been easy. But after carefully looking at different options and considering the requirements, system lifespan and other constraints, we decided to use Groovy-based DSL as the Rules Engine and a simple custom built Groovy-based Workflow Orchestration Engine.
This decision aligned well with refactoring as well. We simply pulled out each of these as separate physical module. It enabled us to focus on managing essential complexity of the business and not fuss around a rigid BPM product.
On the process side
Architectural Reviews
Architecture, as I said earlier, is about making big decisions early on and under given context and constraints. But certain times we may be ignorant or blindsided by our own presumptions that we assume what we are doing is just and appropriate.
This is why an internal or external architectural audit makes sense. You will be asked straight and honest questions about the choices you made. It will force you to rethink your previous assumptions. It will also make you analyze the areas of design and architecture that did not receive enough attention.
If nothing else, it will bring the development team, reviewers and stakeholders in the same room to discuss design and architecture and create a shared understanding. This in itself is a big win. It will make you realize and accept shortcomings and arrive at a plan to address them.
We went through three separate rounds of architectural review before the refactoring. They helped us to discover areas of weakness and strengthened the case for restructuring.
On the people side
Stakeholder Trust
The most important thing for any long-term project is the trust of stakeholders in your venture. This “trust” is not a magical thing that happens overnight. It takes hard work and long time to earn. Once you earned it, it requires even more work to sustain it.
You have to continually provide real value, in terms of real dollars for the work you are doing.
Now imagine a venture where direct and immediate benefit of a restructuring exercise is very small. Most of the benefits are long term. You can quote all the projected ROI all you want, but unless you earned the trust, no one will believe you and restructuring effort will stay on the ground.
Fortunately, our team had the right amount of trust, not less, not more, just the right amount. From the primary sponsors, to the business to the implementation partners, they trusted us, but kept us on the toe.
In my opinion, stakeholder trust is a must in any risky venture.
Implementation Team
“Wars may be fought with weapons, but they are won by men” — General George Patton Jr.
I can’t say it loud enough — large-scale restructuring is effing hard. The only way you can pull through such an arduous task is when you have rock solid team that is smart, well rounded and relentless.
You will need these magic ingredients –
- Willing to unlearn what they have learnt for years
- Willing to learn new ways in the shortest time possible, fail at it, recognize it and learn it again
- Trust their leadership on the intent but work as a team, provide suggestion and actually solve it in a way better than it was prescribed
- Made up of team members with complementary set of personalities and expertise.
I am proud of my team. You rock.
What’s coming next?
Next article is cover what was actually done, the process behind arriving it, challenges and learnings along the way.
To read it when it appears, please subscribe to this collection and follow me on twitter.
If this topic interests you and have more questions, I would be happy to discuss further. Please reach out to me.
