We’re Agile! Why Do We Need Architects?!
Part 2 of 2: Where does a software architect fit on an agile team?
Part 1: What is Software Architecture?
Many existing Agile frameworks don’t consider the role of an Architect in the methodologies, instead simply blending it with “Developer”, or “Person who builds the product”. On the surface, Agile principles do seem to conflict with a traditional software architect role. By this definition, the team must be multidisciplinary and must therefore collectively have all the skills necessary for the creation of the product. In an agile workflow, where the creation of working, validated code takes precedence over nearly everything else, there is no clear separation of architecture from design and implementation. They are all bundled together in the practices used to produce the code. Instead of traditional documents and diagrams, the architecture is going to be exposed through the software itself. Code as documentation. What could go wrong?
Why is this role needed at all?
What happens when architecture is ignored, poorly executed, or generally not supported or regarded as a primary activity within the lifecycle processes?
We’ve all seen the results.
It’s conceivable that for small, well-defined, or frequently-built applications, such as your basic data entry application (“text boxes over data”) with minimal or trivial non-functional requirements, you could get by without an Architect, provided that you have strong or “mature” Developer skills on the team. However, too often, the architecture grows incidentally or accidentally, with too much emphasis placed on behavior over structure, and little regard to critical quality concerns related to the code and structure as a whole. This is the genesis of technical debt.
Over time and repeated iterations taking shortcuts (due to time, or simply lack of recognition/skill/interest), this ultimately results in an eroding architecture that becomes too expensive or unstable to continue to maintain, and the word “re-write” begins to be expressed. Notably, a re-write won’t solve the problem if the core issues that led the team into that predicament are not acknowledged and properly addressed. This concern becomes apparent to the development team working in the code well before it becomes a glaring issue for the product management team. Hence the importance of the technical team as stakeholders alongside the business representatives, in order to develop that shared understanding and reduce silos of information.
The signs of this deterioration are clear: it becomes more and more difficult (takes more time) to add seemingly simple functional changes to the application. Developers complain about areas of the system nobody wants to touch, lest the house of cards topple. There is no way to easily verify a change. A change in one area of the system creates an uncontrolled ripple to seemingly unrelated areas. The system becomes non-performant, and nobody has good insight into why, or how to fix it. Costs of each sprint begin to climb exponentially and adding people to the project only slows it down further. It ultimately results in a general inability to adapt or quickly react to any internal or external influences, which is the whole premise of Agile!
Whether or not you have a named (or skilled) Architect on the team, architecture happens. Every software system has an architecture, whether deliberately chosen or not. Without sufficient attention given to the important quality attributes beyond the code, there will nearly always be a penalty down the road, if that product is continually modified, extended, or shifted into new dimensions to meet customer needs. As systems grow more complex, this skillset becomes increasingly crucial to the success of the product.
Bringing it back to Agile Architects
Based on past practices, there can be stigma attached to an Architect’s role on a project. Historically, they have been called non-pragmatic, that their views of the system design are too idealistic, complex, expensive, impractical, etc. In Agile methodologies, project roles and responsibilities have aligned to work in new ways, by means of greater collaboration. The role of the Architect is no different. Traditionally, Architects provided up-front guidance about system design, functionality, interfaces, technology, etc. and were less involved during the build phases. It is now common for Architects to stay hands-on through the remaining iterative phases of a project.
Dave Thomas has said:
Big design up-front is dumb.
Doing no design up-front is even dumber.
Following an Agile framework, most of the work that an Architect does needs to happen before a set of backlog items gets turned over to the development team; however, each Developer also needs to understand the system’s design and architecture before any code is committed to the production branch to complete a story. This means that, ahead of assigning backlog items to the team, the Product Owner needs to work with an Architect to ensure that these concepts are accounted for in the forward-looking design. The Architect’s role is therefore one of the few (along with the Product Owner) that needs to have a long-term vision of the product, beyond the next couple of sprints.
By nature, Agile projects tend to focus only a sprint or two ahead and Developers therefore aren’t forced to think through their sprint deliverables in relation to the system, and the impact to its extended ecosystem. The Architect directs sprint-level technical discussions and makes adjustments to steer the team toward the enterprise and/or system architecture needs. This includes revising the solution architecture documents regularly to act as a reference point for the development team during the build-out. Without this reference point, a series of smallish solutions viewed in context a single user story, can quickly devolve into an unwieldy mess. This problem arises out of the agile philosophy of doing the simplest possible thing to satisfy a given story, and not trying to solve future problems today. This works great… except when it doesn’t.
It takes skillful judgement and discipline to get this balance just right. Agile teams face the risk of building too tactically, or towards incorrect, unmanageable, or out-of-date system design. Letting your architecture emerge at the speed of the next iteration is simply not feasible when you are building large, complex software systems containing many moving parts, dependencies, challenges, and unknowns. It’s the Architect’s primary job to observe in real-time and redirect before too much time and effort is spent in the wrong direction, and then involve the Developers in solving for those issues.
Without appropriate architectural investigations and on-going attention, it is nearly guaranteed that the project will veer off-course.
What gives the Architect the insights that the other team members don’t see? This does not suggest that the Developers cannot or will not identify these risks as well, but the Architects’ focus is different. They are constantly viewing the system from a different perspective than the rest of the team; they usually work across projects or system areas and spend their time at all levels of the project. They interact often with customers and analysts, product management and the executive team, so also have greater insights into other related systems, allowing them to help juggle competing enterprise priorities, streamline processes, facilitate reuse of software solutions, etc. While the team is busy in the coding details, which itself requires a great deal of focused attention to do well, the Architect is thinking about how to tie the user story into the higher level architecture of the software and, importantly, how to implement just enough of that architectural vision to get the current story completed and into the customer’s hands, while still moving the software product toward that final vision. This ability to guide individual projects towards the larger organizational goal will provide great benefit, by increasing harmony and reducing wasted cycles across the organization.
The Architect makes sure each technical team member knows the higher-level concepts behind the architecture, and what qualities they should be looking for in the solution, to match the architecture vision. They do this by working with the team and staying involved in all aspects of the project.
Tying back to the sentiments of the Agile Manifesto: a “self-organizing team” doesn’t necessarily need a dedicated Architect. In that case, every Developer needs to be able to zoom in and out of the implementation detail, and work at the architecture level when necessary, and then communicate those adjustments accordingly.
In reality, despite best intentions, truly self-organizing teams aren’t all that common.
Much of this could be attributed to the way team members rotate onto and off of product teams. Despite aspirations to be agile, a “collective ownership” of the architecture role across the team is likely going to result in architecture considerations getting ignored or overlooked for short-term gains in velocity. The teams that lack this ability need a more direct strategic leadership approach and they will benefit from a dedicated Architect for the technical and visionary aspects of the system. It also provides a point of contact that can effectively represent the technical team’s needs and interests to the business, so it is accounted for in decision-making at that level. Often, the stakeholders and managers do not have a technical background to understand the complexity of the project and the importance of good architecture. The tendency is to believe that the Developers can just keep “coding away” to deliver the project. This too works great… until it doesn’t. The Architect provides this glue.
Agile or not, the software architecture role exists. In the end, the goal is to find a path towards more intentional architectural thinking and practices. Having a role whose focus is to objectively evaluate the longer-term implications and consequences of immediate choices to solve for “simplest possible thing”, and that isn’t constantly under the same pressure of due dates and individual productivity, is important to provide guidance toward best enterprise state. You are nearly always going to get a better result from a skilled individual with ownership responsibility, than from a collectively owned process where everyone (or quite likely, no one) is looking out for those interests. The result of not doing so, can be very costly to your business.
