Fixing Agile development. Part 2.
It doesn’t scale unless it is aligned with the architecture.
The key requirement in maintaining a team’s autonomy (and hence its agility) is minimizing cognitive overload caused by redundant communication mechanisms that disrupt the flow. In my previous article, I highlighted an issue that makes Agile adoption difficult: companies emphasizing customer collaboration via Product Ownership and neglecting feedback from working software. Not because they see it as unimportant but because it requires a CICD platform built and tailored specifically for the team. Team Topologies book introduces Platform Teams and Collaboration, which can aid CICD adoption.
This time, I want to tackle the issue of coordinating multiple Agile teams. Some believe it doesn’t scale. I have a different opinion. Of course, the Agile team does not scale vertically (increasing team size) by design. So we have to maintain multiple Agile teams. It comes with some coordination overhead, like any other distributed system.
The problem we want to solve is the best allocation of work across multiple agile teams so that their communication and cognitive overload are minimal.
First, let’s explore some real-life anti-patterns that cause unnecessary inter-team communication.
Anti-pattern: Agile Functional teams.
The first anti-pattern stems from the attempt to apply the Agile process in the wrong team structure. It is still common to have teams divided by function (skills), like Frontend, Backend, Analytics, Data Science, etc. It makes sense from many viewpoints. Engineers with the same skillset can collaborate on hiring, best practices for a specific technology, code review, etc. All of these are non-product concerns and hardly need an Agile process. When Agile methodology is accepted mindlessly, it will result in Agile teams like these two: Agile Analytics and Agile Data Science. These examples are real because Analytics and Data Science were often seen as separate domains. Both work with data produced internally. And it took a long time till we started treating data as a part of the product (Data Mesh), and the process is still in its early adoption stage. Agile functional teams cause situations like this one:
It is easy to see the coordination effort between two product owners, which makes the whole feature development less agile. Alternatively, teams themselves may cooperate, but again we will effectively get one large team instead of two autonomous ones. Approaches like SAFe were designed to facilitate such collaboration. But SAFe was criticized as an attempt to put fake agile in the wrong organizational structure. Or, as Dave Farley notices that by design, it adds requirements of time and predictability, contradicting the original Agile manifesto. SAFe doesn’t have reduced communication as one of its goals.
The bottom line is that Agile Functional teams are a poor way to scale work in organizations trying to be Agile. And SAFe is just an attempt to mend an already broken structure.
Anti-pattern: Agile Cross-functional teams.
There is nothing wrong with the team being cross-functional. It has to be universal, and having diverse skills is a plus. The problem starts when you apply team-first thinking. It leads to the following idea:
Ok, we have built three universal Agile units. Let’s assign them our next three projects.
The factor that defines who is assigned the next project is the team’s availability, nothing else. Projects are seen as transitory, and teams must switch contexts from one project to another. It may work in a water-scrum-fall, the broken Agile approach I mentioned. When you try to adopt real Agile, you’ll inevitably see that it is much better to develop a sense of ownership and keep teams working in the same context — the bounded context.
So, we want to have cross-functional Agile teams. But we want to ensure little to no coordination between them. In other words, to apply the architectural concept of modularity (low coupling, high cohesion) to groups of people. Yes, architecture and team allocation are connected. I think the famous quote from Michael Nygard is relevant here:
Team assignments are the first draft of the architecture.
And to complement it — another one from Ruth Malan:
If the architecture of the system and the architecture of the organization are at odds, the architecture of the organization wins.
So, if the teams are designed around department structure (e.g., Agile Analytics team), architectures will follow (siloed Data Warehouse) unless we do something about it.
Hence we want to design frictionless (in terms of communication between subsystems) architecture, and that is how we discover Domain Driven Design.
Intro to Domain-Driven Design.
Readers familiar with DDD may skip this part and scroll to the next section. I will briefly overview Strategic DDD concepts and apply them to team allocation in a real-life architecture.
Definitions.
Domain (of the software) — a sphere of knowledge, influence, or activity. The subject area to which the user applies a program. It could be further divided into subdomains by discovering unique characteristics inside the domain.
Model — a system of abstractions that describes selected aspects of a domain and can be used to solve problems related to that domain (including abstractions in the programming language).
Context — the setting in which a word or statement appears that determines its meaning. Statements about a model can only be understood in a context.
Bounded context — a description of a boundary (typically a subsystem or work of a particular team) within which a model is defined and applicable.
Ubiquitous language — a language structured around the domain model and used by all team members within a bounded context to connect all team activities with the software.
DDD starts with tech management, which needs to differentiate the core subdomains of the business (competitive advantage) and prioritize them. Generic subdomains represent well-known complex problems someone else solved (hence buy or adopt existing solutions). Supporting subdomains are essential for the business to function but will not give a competitive advantage (therefore, outsource the development, if you can).
The Process.
Design step number one is interacting with domain practitioners (“business”) to understand the domain.
The goals of such collaboration are:
- Split each core subdomain into smaller subdomains (some generic or supporting).
- Develop and maintain a ubiquitous language across one or several subdomains, a set of definitions and artifacts understood by both domain experts and developers. Use this ubiquitous language to draw a bounded context, a set of subdomains that speak the same language. Make this separation clear and make bounded contexts independent of each other.
- Organize development projects (and teams) around bounded contexts and strive for their autonomy.
Once bounded contexts and teams are identified, they work in iterations of
- Discovery (building and updating mental models from domain knowledge).
- Design (building solution model from mental models).
- Implementation (of solution models in code).
Techniques like Event Storming exist to facilitate discovery and design stages. One can think of it as long planning sessions in our Agile process.
Contracts.
While teams work on bounded context, we want to ensure they don’t spend too much time running back-and-forth communications with each other. DDD formalizes several contracts between bounded contexts, depending on the situation.
Partnership — changes are coordinated in an ad-hoc manner between two contexts. This approach works if one team supports both bounded contexts.
Shared Kernel — rather an exception to the DDD approach, is a shared codebase developed by several teams. It is usually required when API is hard to define or functionality is too expensive to duplicate in two contexts, so teams choose to share the ownership.
Several customer-supplier contracts:
Anticorruption layer — the consuming bounded context implements a thin layer, which ensures the input to the rest of the context conforms to the expectation.
Open-Host service — the producing bounded context exports published language (not exactly its ubiquitous language), so consumers always get what they expect.
Conformist — the consuming bounded context changes every time the producing context changes. It could be the only option if a producer is complex and business-critical.
The breakdown into bounded contexts with their modes of interaction is called Context Map.
Ok, it was a whirlwind tour of Strategic DDD. I hope readers can catch two important details: autonomous bounded contexts and predefined contracts between them.
We’ve already allocated our teams in the process of DDD. But let’s do it with an example.
We will explore a Context Map of a hypothetical DMP company. We have already identified bounded contexts.
It starts with Data Acquisition. Data comes from third parties through different channels and formats (JSON over HTTP, CSV file, etc.). Details may differ for each partner, but the Data Acquisition context will standardize all inputs and expose them through Open Host Service to other bounded contexts. So internal clients get a stable API, while the Data Acquisition team handles integrations with new customers. The bounded context hides all the complexity of partner interaction (and its ubiquitous language) and only exposes its published language (API).
The Data Marketplace context oversees data aggregation, storage, and preparation for the buyers (and other concerns, like data privacy). It consumes standardized data via OHS. The business is organized so that DMP needs to conform to the buyer’s rules (data delivery mechanism, format, etc.) The Data Marketplace team resorts to conforming to the Sales bounded context — every time there is a new buyer for the data, the tech team will have to integrate with its API.
The Sales team works closely with analysts experienced with the third-party CRM system, so both contexts have a shared kernel.
The Data Marketplace team shares transaction records with Analytics, which built a special Anticorruption layer to get information from these transactions. ACL manages data cleaning and standardization since these transaction details may change and contain some internal or restricted information. The Analytics context uses a third-party Data Lake solution and presents statistics as SQL tables. Table definitions are agreed upon in advance and never change; hence they serve as an OHS layer. CRM queries this data with SQL.
The Data Enrichment team, on the other hand, is always experimenting with different AI approaches, and they not only use standardized data but raw transactions and intermediate results from Analytics. That is why they are partnering with Analytics Context. Finally, it produces more data for the Data Acquisition context in the agreed-upon format (OHS).
Many services depend on the Identity Management solution, a third-party SSO service with a defined API (OHS).
In the picture above, Data Marketplace and Data Enrichment are core domains. Data Acquisition is a Supporting domain; the rest are Generic (all use third-party solutions).
Armed with the context map, we can propose technical team allocation to bounded contexts, developed in-house: Data Acquisition, Data Marketplace, and Data Engineering teams. Teams interact via well-defined contracts. Occasional contract refinement might be necessary, but it requires minimal interaction compared to consistent collaboration between two teams. Note that one team can cover several bounded contexts, especially when the partnership is used. At some point, the decision could be made to split Analytics and Data Science (for various reasons, like team size), then ACL or OHS should be designed between them.
Conclusion.
I was looking for ways to split work between Agile teams that enables their autonomy and hence improves their agility.
Aligning teams with high-level architectural building blocks is a good strategy. When it is neglected, it leads to Agile team allocation anti-patterns.
This approach only works if we provide well-defined contracts between architectural components.
Domain-driven design is one of the options the company can use to facilitate modularity. It requires some governing body (Architects & Domain experts) who identifies bounded contexts and decides team allocation around them.
