A case for Agile
Over the last years there has been a proliferation of Agile methodologies and frameworks which claim to solve the issues and shortcomings of their kin. If any of these claims were completely true, we would expect this multiplication of methodologies to come to an end, but that doesn’t appear to be the case. While there is likely value in each of these methodologies, none of them can be universally successful. Any methodology that claims to be the Agile Project Management silver bullet is likely missing the point. In fact, I think that, to a certain degree, many are. While I don’t claim to be an expert in Agile, I do believe that the intention of the Manifesto for Agile Software Development was not to replace pre-existing processes with other processes. In my interpretation, the intention of the Manifesto was to change a rigid and stagnant software development mentality that was hindering evolution in the industry. It was aiming at a shift of focus to enable a spirit of exploration and discovery, to find better ways of developing software and delivering value. This open minded way of thinking, this will to question the status quo with a fail fast and learn philosophy has enabled some of the most valuable innovations that the software development industry has seen in years. It is in trying to apply a rigid structure disguised as an expression of the Agile Principles that “Agile” methodologies fail to understand the thought process that inspired the Manifesto. Any fully defined methodology unwilling to adapt on a case basis and evolve by questioning its structure is not an expression of the Manifesto. But any methodology willing to adapt and evolve is, by definition, not fully defined. Agile inspired practices and ideas should not come in pre-packaged sets, they should be emergent behaviours of experimentation and adaptation, shared products of a community of interacting and self-organising teams that recognise themselves in a never-ending journey of improvement via trial and error. This is, in my opinion, the behaviour that most adheres to the spirit of the Manifesto and that is most likely to enable exponential growth and innovation. The values expressed in the Manifesto are still very relevant today and I think it’s worth trying to understand them before exploring any Agile methodology.
Individuals and interactions over processes and tools
Processes are useful to support humans in tasks they are not naturally good at. They provide a repeatable set of steps which can be used to analyse and consolidate successful experiments. They should be sufficiently broad to allow evolutionary variation, yet detailed in low risk tolerance aspects. Excessively detailed and strict processes limit the freedom of action that humans need in order to experiment and improve. Excessively loose processes in critical areas don’t help minimise risk beneath an acceptable threshold.
Humans are particularly good at understanding context, recognising patterns and inferring knowledge. They have the capability of maximising learning through interactions by sharing information and experiences. They are limited in learning when confined to mental spaces that don’t engage and challenge their intellect. Humans are not good at remembering and repeating exact steps, they are closer to adaptive processes than to deterministic machines. Computers, on the other hand, don’t understand context and have limited inference capabilities, but are particularly good at reliably delivering repeatable outcomes by following detailed and unambiguous instructions. That is why aspects of a process that, due to risk aversion, do not allow for variation, should be automated. Processes and tools are extremely valuable when used to support individuals in their interactions and their natural capability of adding value by discovery. Delegating tedious repetitive tasks to automation allows individuals to focus on learning and evolution. Processes should therefore be implemented in ways that do not treat humans like machines, but trust them in their abilities and support them by minimising the necessity of directing efforts in machine-like tasks.
A successful process applied to a rapidly changing environment is likely to have diminishing returns over a relatively short period of time unless given the opportunity to adapt and evolve. The success of a process is influenced by circumstances. The software development, and, more in general, the technological industries, are evolving at exponential rates. Remaining innovative becomes increasingly harder unless we are given the right set of tools and environments to maximise and accelerate our natural capability of learning and adapting. That’s why it is fundamental to continuously challenge our assumptions and adapt our behaviour by retrospectively reviewing the failures on which we built our achievements. I believe that the best way to envisage and materialise successful visions is to rapidly achieve milestones through small iterations. These allow adjusting course of action at minimised costs. Iteratively adaptive processes are more likely to stand the test of time than rigid and immutable methodologies due to the rapidly evolving nature of the industry.
Working software over comprehensive documentation
Some may argue that comprehensive documentation is required in order to minimise the risk of mistakes. This way of thinking presents fallacies that are evident when challenging the assumptions on which it is based on. The first assumption is that the documentation is correct. But, because documentation is written by humans, it is prone to errors. The second assumption is that developers will follow the documentation to the letter. The more information dense the documentation, the more likely a developer is to miss or misinterpret parts of it. The third assumption is that the solutions described by the documentation can be closely translated into technical implementations. But, more often than not, it is hard to predict technical limitations that may require to diverge from the documentation in order to implement a solution. These problems are amplified when information is serially passed on and deprived of context by the time it reaches the development team. Developers who don’t understand the problem that a solution is supposed to solve are less likely to produce outcomes that align with the objective. Without context they are less likely to notice documentation errors. Without context they are less likely to avoid mistakes. Without context they are less likely to circumvent technical limitations without diverging from the objective. When mistakes are finally identified, amending them likely becomes a lengthy and costly process. The fourth, and possibly the most important assumption, is that clients can clearly convey what they need. A client may be able to articulate in a very detailed way what he or she wants and technical teams can be very effective at delivering exactly what the client requested, but this may not be what the client actually needs. Often a client is trying to solve a problem and has a solution in mind, which is conveyed from his or her point of view to the business, but the provided information may be insufficient to identify the underlying problem and envisage a technical solution in its entirety. It is possible that the required information may not be available upfront and may only be achievable through investigation. When this is the case, comprehensive documentation is not feasible from the get go, which can hold back a project, or worse, require to fill in the gaps without experimentation in order to proceed, which may lead to costly errors.
A far more effective, less error prone approach, is to minimise information overhead and rapidly implement prototypes for immediate feedback. An iterative process of incremental changes and corrections based on client feedback is likely to direct development efforts toward the optimal solution. Continuous interaction between client, business and technical teams allows each party to contribute with their unique perspective toward the definition of a solution that best satisfies the client’s needs. Working software also allows for instant client gratification and reinforces positive perceptions. All of the above contributes to the idea that this kind of approach is more likely to deliver the best value for the client at the lowest cost when compared with non agile alternatives.
Customer collaboration over contract negotiation
Requirements are hard to gather upfront. Client requests, especially at the beginning of a business interaction, can be symptomatic of a problem, but may not constitute a solution to the problem itself. Trying to agree on the entire scope upfront minimises the chances of identifying the underlying problem the client is trying to solve. This, in turn, increases the possibility of ultimately delivering something that is not of value to the client. A group of interacting humans with different perspectives is more likely to converge to the correct understanding of a problem, and subsequently to the identification of a solution, than a single individual or group sharing the same base knowledge and perspective. People with different perspectives enrich the conversation and offer points of view that favour the discovery of nuances and a deeper understanding of fundamental aspects of problems and solutions. A business analyst can help translate the client’s requests into a viable solution, a developer can offer a technical interpretation, estimate feasibility and pre-empt technical issues, the client can in turn provide feedback on the solution and its alignment to his or her needs. Iterative interactions interleaved by experimentation maximise the chance of converging toward a solution that delivers the highest value. Initial interactions between client and business should be focused on collecting sufficient requirements to implement a prototype. As soon as ready, this should be demonstrated to the client in order to receive feedback. Changes and amendments should be made accordingly and the cycle should be repeated until mutual satisfaction is achieved. Prototypes and examples can be very effective in providing both the client and the business an understanding of what is actually needed. Alignment to success is far easier to measure on concrete examples than on documentation. Rapid feedback allows adjusting course of action toward an objective that becomes clearer with each iteration. This approach is not only effective, but also efficient, as it minimises efforts in the wrong direction. Small changes with continuous feedback reduce costs and allow reaching a solution much faster than an implementation strategy based on requirements gathered upfront which may be hard to understand and ultimately incorrect.
Another important aspect to consider is trade-off analysis. Clients want more for less, they desire an optimal solution at low cost to be delivered in short time frames.
As experience demonstrates, quality, cost and time form a triangle in which each aspect represents a constraint. For example, maximising quality at a fixed cost requires more time. In practice, though, trading between constraints is not always possible. Allocating more resources to a project, can, under certain circumstances, be counter-productive by leading to reduced quality and increasing time frames. Communication overhead and ineffective division of labour can introduce errors, which then require time to be addressed. Project management is ultimately the art of balancing constraints by verifying assumptions, adapting through experimentation and learning from experience. One of the greatest challenges is achieving a balance between scope, budget and time frames that satisfies the client’s needs. Trade-off analysis applied to agile iterations aims at repeatedly finding this balance for each milestone. At the end of each iteration the goal is to deliver something of value. Rapid iterations and continuous feedback minimise the costs associated with changes, as these will be smaller the more frequent the feedback. At each iteration, the client and the business should agree on which features should be prioritised and which can be postponed or even dropped. Eventually these iterations should lead to a Minimum Viable Product, i.e. a sub-optimal solution that addresses the problem to the point of being usable in a production environment. This solution can be improved through subsequent iterations, and, while not being ideal, has the great advantage of delivering value to the client sooner than waiting for the optimal solution to be ready. Also, releasing a Minimum Viable Product to the end users can provide a measure of success and insights on what the optimal solution should actually be.
Responding to change over following a plan
In order to adjust course of action according to the feedback provided with each agile iteration, developers must be able to implement changes rapidly whilst minimising the chance of errors. This requires engineering practices that embrace change at any stage of development and maintain high quality standards. Design patterns, architectural patterns and software design principles have traditionally emerged as a response to common development problems. They facilitate future changes and improve overall quality, but they are not an ultimate set of solutions to all software development problems. Rather than blindly applying these patterns and following these principles, it is fundamental for developers to understand the mindset behind these practices. This allows for them to conceive novel and unique solutions to the problems they face in the same spirit of maintainability, extensibility and high quality standards. I have identified software design practices that have improved my career and I encourage every software engineer to experiment and decide what works for him or her. Not only, I encourage every software developer to continuously review his findings and test them against his or her current knowledge in order to avoid stagnation. This way of thinking should ultimately move from an individual to entire software development teams in order to favour evolution.
Development teams should be given the opportunity to self-organise and evolve. It is far more effective to provide a clear vision and direction to development teams than to micro-manage them. Good developers excel when given the opportunity to experiment, learn and adapt. As long as the goal is clear, they should be able to envisage the best solution based on their experience and identify steps which can be generalised and repeated. In their software development practice, they should be able to detect the different rate at which software components change and organise dependencies in a way that couples components with similar change rates. This minimises unwanted side-effects and makes reasoning about software and changes simpler. Developers should be able to discern general responsibilities from specific applications, inferring and consequently identifying patterns to which solutions can be re-applied. They should be able to expand and share their body of knowledge, increasing the overall quality standard of the team. They should be encouraged to share knowledge and mutually stimulate their passion. They should be guided toward regularly reviewing their practices and challenge their assumptions in order to favour evolution. They should be provided more questions than answers, as this engages them, provides meaning to their work and ultimately compels them to maximise their efforts whilst minimising waste. This approach helps build a culture of trust. Interactions based on trust remove the necessity for strict rules and regulations. People start sharing goals instead of propagating orders and start relying on each other to get the job done. The best talent for agile environments is usually attracted by a culture of trust. In a culture that is enriched by the best talent, leaders who trust their teams with a clear vision can rest assured that people will do what they do best to contribute toward the overall achievement of an objective, while also learning and improving during the process.
Failure is an option, as long as its used as a means for learning. Mistakes are welcome, as long as they are openly and shamelessly shared and used to improve. Every failure is a step toward success, just as every achievement should inspire to go the extra mile next time.
Ultimately, agile teams demonstrate their value in their capacity to deliver value effectively and efficiently, i.e. by understanding problems, envisaging solutions and rapidly adapting to change. A team enabled to evolve can drive innovation and constitute a strategic advantage for the business.
Conclusion
How do we measure success? By the value we provide through our actions, by how effective and efficient we are in doing so and by how enjoyable is the whole process. Success is incomplete unless all parties at a table are satisfied.
How can we achieve success? Different perspectives can contribute to understanding problems and converging toward solutions. Rapid iterations of experimentation and feedback allow incremental delivery of value and minimise risks. A culture of trust encourages exploration and evolution. Engagement and achievements promote team happiness, which, in turn, leads to more engagement and achievements, resulting in a positive self-reinforcing feedback loop. These can be very effective and efficient means of achieving success.
The ideas that inspired the Manifesto for Agile Software Development and the values and principles it presents are valuable guides for achieving success and driving innovation. They are not meant to be translated into rigid processes and presented as complete solutions, they are meant to challenge and inspire a journey of continuous improvement, where failures and achievements are just steps toward excellence.
References & Further Reading
Manifesto for Agile Software Development
Miell, Ian (15 October 2017) My 20-Year Experience of Software Development Methodologies, Online Article
Jorgensen, Greg (10 February 2014) Why don’t software development methodologies work?, Online Article
Cutler, John (5 October 2017) Why Isn’t Agile Working?, Online Article
Stagner, Dave (12 December 2017) Iterations, Not Sprints, Online Article
