Fixing Agile development. Part 1.
A single team can’t do it in isolation
Agile creators declared it dead a long time ago. But it is not the message I want to reiterate. It has drifted too far away from original ideas, and plenty of articles are written on the topic (start from Martin Fowler to get the gist). I have a simple (and probably simplified) explanation of why it happened. We had a strategy (Agile Manifesto), but the execution was delegated to the public, with little restriction and a low entry bar. There are powers at play (capitalism) that will focus on optimizing for profit, relentlessly trying to entice more customers. In the end, these powers may change the course of execution. And there is no counterforce to correct it on such a scale. Ironically, Agile adoption is not agile. I spoke about these forces in my other article.
Nothing can be agile on an industrial scale. We usually think of the Agile approach in the scope of a single project. There are ideas out there to bring agility to the scope of one enterprise. But in a whole industry, when multiple players are competing with each other, it looks too far-fetched (if ever possible).
Anyway, in many instances, the Agile process is broken. As someone who has been practicing it in different setups, I agree with that. We must find ways to nudge it in the right direction. I will do this by combining it with other bright ideas.
The Team Topologies book is full of such ideas. It describes ways to organize multiple teams in a modern enterprise driven by DevOps practices. If I have to summarize it in one sentence, it would be “autonomous teams optimized for flow.” It deliberately stays away from “Agile,” but will it help to think of Team Topologies and Agile as two pieces of the same puzzle? Agile started as a bottom-up approach, created by engineers for engineers. Team Topologies takes a different starting point by looking at the organizational chart and Conway’s law. Both aim to improve the development process. I will examine the connections between them.
Evolution of Agile.
We need to start with Agile. I will not delve into implementation details here. Instead, I’d like to explore the mechanics of the software creation process, when several people need to coordinate their efforts to achieve the best outcome. The Agile approach is an alternative to the traditional continuous operation production flow. This software development process:
While specifics may vary, the fundamental concepts remain the same: specialized individuals or teams — let’s refer to them as roles — work on production stages and produce artifacts, such as documents or working software. Once a phase is complete, the team passes the artifact to the next unit in the operational flow (I like how this book describes this staged process).
This conveyor works effectively in manufacturing, where artifacts are produced exactly as specified, and there is no room for uncertainty. I’d like to draw a distinction between manufacturing and software engineering based on the Cynefin decision-making framework, which offers five contexts or domains — clear, complicated, complex, and chaotic — in order of increasing uncertainty. A manufacturing conveyor in a factory is an example of a clear domain. Best practices and standard procedures characterize it. Even during a power outage during the operation, there are guidelines for relaunching the production pipeline as quickly as possible.
The software development process is not a “clear” domain like a manufacturing conveyor in a factory. It falls under the “complicated” and sometimes “complex” domains, characterized by “known unknowns” and “unknown unknowns.” In other words, the software development process involves situations where we can anticipate problems, but there is no one-size-fits-all solution. Sometimes, we may encounter unpredictable or unprepared situations, making it a complex domain.
The Agile Manifesto recognizes the intricate nature of software development and acknowledges that using traditional methods designed for clear domains are ineffective in software engineering. Instead, it emphasizes the importance of implementing “probe-sense-respond” (from Cynefin vocabulary) feedback loops to tackle uncertainties. Applying this principle to a continuous operation pipeline without considering its unique challenges would result in multiple slow feedback loops:
Right-to-left arrows in the diagram describe feedback loops. And too many arrows hinder the execution (as the number of back-and-forth communications grows).
To speed up the feedback cycle, we can combine several responsibilities inside one unit (Agile team, which I will call a team in this article). The internal feedback loop is faster because of the team’s collocation.
The structure above got the name water-scrum-fall. It is broken. It has slow feedback loops between the Agile team and Infrastructure Engineers, Operational support, and Analysts. Unfortunately, many companies are stuck in some variation of this operational structure.
There are slow feedback loops, which we need to remove. We must bring the analyst into the team (Product Owner), provide a Platform for the team to maintain the infrastructure seamlessly (DevOps, CICD), and introduce the practice of operational support into the Agile Team (make it a Product team).
Product Owner.
It is a relatively familiar role nowadays. It does not follow directly from the doctrine, was criticized by the founding fathers, and is indeed arguable. Let’s look at the pros. Instead of the Analyst preparing a long and complicated specification document, the Product Owner becomes a living representation of the document in the Agile team (how agile is that). One of this person’s key roles is to shield the team from the complexities of business decisions and only request its help when technical expertise is necessary. Otherwise, they are the source of business requirements, and the team is responsible for design and implementation.
On a negative note, the person covering the development team from interactions with the business may eventually become a bottleneck. Developers will lose the sense of the business domain. It is a tricky equilibrium.
Key takeaway: consistent communication with business stakeholders is one of the pillars of Agile. At the start, the whole team can do it. But once the product grows and the development team is loaded with other tasks, like operational support, it seems wise to have a dedicated person to maintain the interaction with business clients. Their responsibilities will also include maintaining a shared understanding of the business across the team (which I have rarely seen in practice).
Platform.
To make deployment and internal feedback loop from pre-production and Production systems faster, it is necessary to give the team tools to deploy and run the application instead of handing it to other specialized engineers.
The practice of CICD came along, which advocates the automation of testing, validation, and deployment (popularized by this book). Combined with other cross-cutting concerns, like networking and security, the practice became known as Platform.
Setting up the CICD pipeline is a complicated endeavor, and it brings us to Team Topologies. The authors delegate Platform creation to a special Platform Team (PT) in the book. So how should the PT operate? Can it be Agile?
First, let’s notice that the main client of PT is the other development team (called the Stream-Aligned Team in the book, further SAT). Platform development takes time, and the PT needs feedback from the SAT. They can only achieve its true agility when the Platform is fully functional.
It all sounds complicated, and the book proposes a way to handle it called Collaboration. It means PT and SAT work together as a single unit until at least the first viable version of the Platform emerges. It sounds obvious to those who read the book, but I have never seen this in practice. Instead, SATs continue running their own Scrum process. PT will have to interrupt this process (at best) or build the Platform without considering what SAT needs (at worst).
It takes deliberate effort (and maybe pausing Scrum or other timeboxed approaches for a while) to combine these two forces for a short term.
Once the Platform is ready, the PT may transition to Enabling Team (ET), which ensures the adoption of the Platform by other teams, again via Collaboration.
Finally, PT should focus on one SAT at a time. That is why developing a Platform that satisfies all SATs may take a while.
I want to pause here and let the reader imagine how it might work in their organization. Let’s say you have DevOps engineers who built the first version of the future CICD platform. Then you invite 1–2 DevOps to join your Agile team and build the pipeline together. You explore different aspects of it, what is possible, and what is necessary for your particular project. Do you need integration tests? Are all components containerized? Is it a streaming application? If so, how do we produce the input for integration tests? If it is a batch application, where does the input come from? Do we want to have coordinated tests with other projects? Allow PT to observe patterns across teams and build shared modules. Do not reinvent the same thing in a silo.
Key takeaway: to be truly Agile, the team must collaborate with the Platform Team to ensure it uses the Platform efficiently. Not many companies got here. Instead, some Agile teams are forced to adapt to a platform that wasn’t designed for them.
Product team.
Once SAT delivers to production, it continues to support the product. DevOps emphasizes practices like observability to aid this. And that is where Agile should end! I mean time-box practices like Scrum here. Unfortunately, I rarely see teams acknowledge that. Instead, they continue running bi-weekly sprints and consuming feedback from multiple places: business feature requests, operational issues, and bugs. Most bugs can’t live in the backlog, so it must be groomed and reprioritized too often. At the same time, new business features aren’t that big and likely won’t need regular practices like demos.
On top of that, if observability is treated seriously, the team gets the most valuable feedback from production metrics. Optimization of the product based on metrics is the essence of Lean practices. It is what the SAT should look at after product delivery.
Key takeaway: The product team does not have to be Agile. There is no need for fast prototyping anymore. Focus on removing waste. Report it to the business (via the Product Owner).
Conclusion
Agile is broken but not irreparable. In this article, I show that the team must follow some evolutionary steps to agility.
And most importantly, it can only achieve agility in collaboration. Regardless of the version of Agile you use, be it a Dark Scrum or a light version of XP, the plan of its adoption should involve steps that the team can’t do in isolation. The declaration of Agile values in a siloed team won’t work.
The Agile Manifesto emphasizes consistent feedback from both business stakeholders and working software. Modern Agile practices focus on training Product Owners, which only covers communication with the business. CICD practice is harder to crack, so many got stuck here. Resolving this will enable consistent feedback from working software, and Team Topologies offers a way to embed these practices into the Agile process by introducing the Platform Team.
So, lesson number 1: learn to collaborate with the Platform Team (create one if needed) to improve the feedback cycle from the working software.
Another important issue that I highlighted is that Agile has its boundaries. Sticking to the Agile approach in the late stages of the product is harmful.
Lesson number 2: be mindful of the stages of Product delivery. At some point, Agile might become a set of useless rituals.
As I said, the Team Topologies book offers perspective on multiple issues hampering Agile’s progress. Next time, I will cover another one — scaling.
