On Complexity: Why Your Software Project Needs Scrum
Software (& product) development is never a simple endeavor. Save perhaps for projects that run a couple of days and involve only a handful of people. The vast majority of projects can easily be considered complex. By ‘complex’ I mean: too many variables and potential (emergent) interactions to reliably predict the (near) future. Yet, we seem to suffer from a powerful cognitive bias that makes us discount complexity with statements like ‘it can’t be that hard’, ‘we’ve done this before’ and ‘we already know pretty well what to make’. I often run into people that feel that their ‘project isn’t complicated enough for Scrum’. I strongly believe that these people are reasoning from a very limited understanding of complexity. In this long-read, I offer several models and frameworks (like Stacey and Cynefin) to help understand ‘complexity’ and better understand why Scrum (or some other iterative framework) really is the best approach.
Throw away your plans?
Most people understand ‘complexity’ solely in terms of ‘Knowing what to create’. Take the manager that recently told me that his project wasn’t complex because it involved a re-build of an existing (large) application. But ‘knowing what to create’ is only one of the factors. During our Professional Scrum Master-training we always ask participants to brainstorm as many factors that influence the success of a project as they can. We take only a few minutes for this exercise. Even within this constraint, we always manage to quickly fill up an entire flip chart. I’m sure that if we’d spend an entire day on this exercise, we probably would be able to fill up an entire wall.
To give you an idea of what people come up with (top of mind):
- Communication styles & skills
- Mandate and support within the organization
- Skill-level of the team(s)
- A clear goal and/or vision
- The technical skill of developers
- Number of people involved
- Organizational culture
- Accessibility of stake- and shareholders
- Involvement of the customer and/or end-users
- The volatility of the market
- Time-management skills
- Temporal factors (like how long will the project run)
- Existing codebase (quality, size, knowledge)
- Relationship with suppliers of important components
“Plans are a massive source of ritualistic process-waste by making people spend precious time creating them, reading them and keeping them up-to-date”
There are some HUGE factors on this list. Like organizational culture, goal clarity, and skills. How can you possibly predict upfront how these factors will manifest during a project, and how they will interact? The reality is that in the face of this level of unpredictability, there is no value in creating plans. Plans are a massive source of ritualistic process-waste by making people spend precious time creating them, reading them and keeping them up-to-date. This is not to say that we should not spend time thinking about what we are going to do. We should do it in a manner that fits with complexity. As Eisenhower already suggested:
“Plans are useless, but planning is indispensable.” — Dwight D. Eisenhower
Several frameworks can help us identify which approach fits with a certain degree of complexity. I will describe the most well-known ones; the Stacey Matrix and the Cynefin Framework.
A short note on levels of complexity
We’ll be talking a lot about activities in the coming paragraphs. Activities occur on different levels, from developing a new product (large) to adding a column to a table (small). These activities fall into different domains of complexity. This means that all domains or types of complexity can exist simultaneously but on different levels. This post was written with the product- or project-level activities in mind.
The Stacey Matrix
One way to understand complexity is through the Stacey Matrix, developed by Ralph D. Stacey (a Professor of Management). The model has been presented in many different forms throughout the years, but it primarily states that complexity is a function of the degree to which there is agreement over what is needed (the ‘what’), and certainty over how we need to get there (‘how’).
The Stacey Matrix was developed to help managers determine the complexity of their environment and adapt their style of decision-making. For software development, the Matrix is often plotted along different axes; ‘Requirements’ and ‘Implementation’ (or ‘Technology’). The former is determined by the obviousness to which we know what we need to build (like the product) and what features to implement (‘What?’), whereas the latter is determined by the obviousness of what is needed to get there on in terms of implementation/technology (‘How?’). It should be noted that this adaptation does not really fit with Stacey’s original Matrix. But it does offer a similar conceptual approach to understand complexity within the context of software development, and what approach to use:
- Simple: An activity is ‘simple’ when it is obvious what is needed and how to get there. Take the development of a basic 1-page website based on an existing flyer or brochure. Both what is needed (‘digitize the flyer’) and how to get there (‘some html/css to replicate the styling’) is obvious. The most effective approach is best summarized as ‘just do it’. Take an existing ‘recipe’ or ‘best practice’, apply it and you’re done. No planning is needed, and neither is an empirical framework like Scrum;
- Complicated: An activity is complicated when it is not obvious what is needed or how to get there. I often use the example of a webshop for a new type of products. The implementation is quite obvious here and can be answered with a plethora of existing modules for webshops (like Magento or WooCommerce). But how to best present and market the products will be less obvious. Another example is connecting two systems to exchange particular types of data. What is needed is obvious here (‘connect the systems’), but the implementation is less obvious (‘How do we map the data? And what mechanisms do we use for this?’). It is difficult to pinpoint a single best approach for complicated environments. Some authors state that a planned approach (analyze, design, execute) can work here. But based on my own experience, I’d say that an iterative approach is still preferable over a planned approach. Although complexity is certainly limited here (to one of the two axes), there are still a lot of unknowns. An iterative approach offers a process that can help uncover them quicker and correct course when needed;
- Complex: When both what is needed and how to get there are not obvious, an activity is considered complex. The interplay between variables becomes too complicated to accurately predict upfront, which makes detailed upfront plans pointless. Instead of trying to control and predict the flow of the activity up front, a more effective approach is to use an ‘Empirical process’. In these impossible-to-predict environments, the most reliable information to use for decision-making is based on what has already happened (and not what we anticipate will happen). Therefore we should work in small incremental steps, using every step to determine if we have moved closer to solving our problem or away from it. Scrum is a good example of such a process;
- Chaotic: The upper-right corner is reserved for activities where everything is in flux. In these environments, even the past does not longer offer reliable information for decision-making. A good example if a service desk for a large application or a major outage of a hosting environment. The best approach here is ‘Triage’: prioritize the problems, solve the most urgent one first, and move on from there (rinse and repeat). Although an iterative approach is still an effective approach, the steps need to be much shorter. Think Scrum with short sprints or Kanban with a low Work-in-Progress limit;
A third axis that is often added to the model (not by Stacy himself, though) is the number of people involved. The idea is that complexity grows as a function of the number of people working on an activity. A task that is simple for a one person may be very complex for a group. Picking a movie to watch is easy on your own, but it may become very complex with a group of friends.
Limitations of the Stacey Matrix
Interestingly, Ralph D. Stacey has since let go of the matrix as a means to understand complexity. His primary reason is that he now considers all activities as complex to some degree. Even the most simple activity can become complex because of sheer (bad) luck.
A second issue with the Matrix is that a naive reading of the Matrix can lead to the conclusion that complexity can be reduced with sufficient brainpower. If we just spend enough time on agreeing what is needed, and how to get there, we’ll turn a very complex issue into a simple one eventually. In other words, the Stacey Matrix could be used to reason that we just have to plan and prepare even more extensively in complicated or complex environments. But this is contrary to the point that Stacey was trying to make with his model.
A third issue is that the model only considers ‘agreement’ and ‘certainty’ as factors that introduce complexity. But there are many other factors that introduce complexity, such as communication, the skill of the people involved, organizational culture and pure chance.
The Cynefin Framework
Cynefin is a sense-making framework that was developed in 1999 by Dave Snowden (IBM) as a means to help leaders and managers understand and act on the situation they’re in. He later refined it with (among others) Cynthia Kurtz (2003). The model has since been adopted by the Agile community to better understand and explain why Agile and Scrum are a good fit for software development. Although it can appear more complicated than the Stacey Matrix, it does offer a conceptually stronger argument.
Simply speaking, the Cynefin framework identifies five domains of complexity that are categorically different. In its most simple form, the framework states that there are four ways of making sense of ‘the unknown’. Each domain has its own way to make sense of the situation and requires a different approach to be effective and a different style of management. Although the Cynefin Framework is often presented as a 2x2 model, there are no dimensional axes underlying the quadrants.
- Obvious: An activity is obvious when the problem is well understood and has a clear, established solution. Diagnosing the problem and finding a solution, therefore, requires no expertise and little skill. The best approach is to use a ‘best practice’: diagnose the situation (sense), categorize the problem into the right ‘bucket’ (categorize) and apply the best practice associated with that bucket (respond). An example of this is following a checklist for a frequently-performed deployment or a call center using call scripts. The style of management that works best in this domain is coordination: make sure that people know what to do and how to do it;
- Complicated: An activity is complicated when diagnosing the problem and finding a good solution requires expert knowledge. Expertise and time are needed to ask the right questions (the ‘known unknowns’) and arrive at a diagnosis of the problem. Once known, expertise is needed to pick one of a limited set of solutions. The best approach is to use a ‘good practice’: diagnose the situation (sense), analyze the problem (analyze) and determine and apply the planned action (respond). An example is solving a bug in an application or connecting two API’s. The style of management that works best for this type of activity is cooperation: work with experts;
- Complex: An activity is complex when both the problem and the solution are mostly unknown. In these cases, even finding the right questions to ask is difficult (the ‘unknown unknowns’). Both understanding the problem and finding a solution requires active experimentation until patterns start to emerge. Although a general direction may be anticipated up front, the actual solution is not apparent until it has been found. The best approach is to actively discover patterns (‘emergent practice’): experiment to gather feedback from the environment (probe), analyze the feedback (sense) and respond with another experiment until you’ve reached the ‘complicated’ domain. The style of management that works best for this type of activity is to work together to find the best solution (collaboration);
- Chaotic: An activity is chaotic when both the problem and the solution are unknown. This is a temporary state. Over time, constraints and structure will emerge naturally (even if you don’t impose theme). An example of this is when a large web farm goes down, or in case of a natural disaster. There is little time to think, so focus on finding something that works instead of finding the best solution. The best approach is to apply a ‘triage’: quickly prioritize and solve a part of the problem (act), analyze the situation (analyze) and move on to the next prioritized issue (respond). The style of management that works best for this type of activity is to take immediate action, provide clear and direct communication and restore order to decrease complexity back to the ‘Complex’ domain. ;
- Disorder: The domain at the center of the framework is reserved for activities where the domain of complexity is not known. Snowden added this domain to emphasize that we have to make sense of the environment first, using data and information available to ascertain the type of complexity. If no such data is available or has not been made sense of, activities remain in ‘disorder’. The first priority here is to determine in which domain the activity belongs. Then use the approach that fits with activities in that domain. This also emphasizes that complexity is not some ‘objective’ attribute of an activity; it is the result of a sense-making process;
Knowledge, boundaries, and where Scrum fits in
A key factor in the Cynefin Framework is knowledge. Our understanding of the situation helps to make both the problem and the solution more obvious. Using the recommended approach for each domain (probe-sense-respond, etc) we essentially help ourselves to better understand the problem. This helps us to transition to a ‘lower’ domain of complexity.
Suppose we have a problem that falls squarely in the ‘Chaotic’ domain, like a major outage of a large enterprise application that hasn’t occurred before. We don’t have time to sit back and think about the best solution, so we’ll have to triage the situation and identify the first steps to take. A highest-priority-first, low Work-in-Progress Kanban-like flow can help here. This will eventually transition the problem into the ‘Complex’ domain.
According to Snowden, Scrum is essentially a way to transition from the ‘Complex’ domain into the ‘Complicated’ domain. By breaking down a large problem — like developing a new product — into many smaller problems (the sprints or even individual PBIs) we eventually decrease complexity for the smaller bits.
So by adapting our behavior, we can increase our understanding of the situation. This helps transition problems (clock-wise) across boundaries, from Chaotic to Complex, from Complex to Complicated and perhaps even to Simple. The reverse happens when knowledge is lost or not applied correctly. An example is the deployment of a complicated application. When done manually, the activity can easily be Complex or even Chaotic. But when deployment is heavily automated it can become a Simple activity. The small ‘fold’ or ‘cliff’ at the bottom represents the possibility to regress from ‘Obvious’ straight into ‘Chaotic’. This happens when existing best practices become outdated, but continue being applied. Or when the person responsible for maintaining a key application leaves the company without transferring knowledge. The next time a major issue occurs, people will scramble to resolve a ‘Chaotic’ problem.
The most important lesson to draw from these models is that software development is at least ‘complicated’, probably ‘complex’ and occasionally ‘chaotic’ on the product- or project-level. When there is a no clear understanding of the desired end-state (the problem) and no obvious way to get there (the solution), an iterative approach is necessary to help clarify both the problem and the solution. No matter how much brainpower we apply to predict the future, the vast web of factors that influence the success of our work makes such predictions all but pointless. Therefore, using a planned approach in a non-simple environment will most likely cause you to solve the wrong problem or apply the wrong solution.
In complex environments, the best source of information to inform our decisions for the future lies not in predictions of that future, but in what has already happened. This means that an empirical, iterative process is needed. Much like how a scientist goes about testing a scientific theory, or how an inventor incrementally works through progressive prototypes until arriving at a working model. Frequent inspection of an evolving ‘current state’ — as aspects of the problem are being solved — is needed to increase our understanding of the problem and the solution, and to inform our decisions about what is needed in the future. By breaking down the larger issue into many smaller problems (sprints), we’re essentially transitioning from the ‘Complex’ to the ‘Complicated’ domain.
When there is a no clear understanding of the desired end-state (the problem) and no obvious connection on how to get there (the solution), an iterative approach is necessary to clarify both the problem and the solution.
So what about Scrum?
In very practical terms, Scrum is a framework that offers such an empirical process. This is exactly why Scrum is not a ‘methodology’. A methodology is a complete set of principles, but Scrum lays down (only) the foundations for an empirical process applied to software development. This requires transparency, inspection, and adaptation. Scrum requires the implementation of five Scrum Events to maximize transparency, allow inspection and facilitate adaptation. Not implementing one of these five events severely limits the quality of the feedback we need to inform our decisions for the future. Scrum is by no means the only option. But it is a very popular option with a large community of practitioners. A good reason to give it a try if this post convinced you that your project may be in need of an empirical approach.
Update (March 23, 2017): Dave Snowden responded to my post on Twitter. He emphasized that Scrum helps to transition from the Complex problem into many Complicated problems (breaking down work into sprints, then into PBIs). A Triage-like process in a Chaotic domain helps to transition the problem from ‘Chaotic’ to ‘Complex’. I have updated the post accordingly, and am thankful to Dave for his personal input!
- Snowden, D. J. & Boone, M. E. (November 2007). “A Leader’s Framework for Decision Making”. Harvard Business Review, 69–76. PMID 18159787;
- Snowden D. J., & Kurtz, C. (2003) “The new dynamics of strategy: Sense-making in a complex and complicated world”, IBM Systems Journal Vol 42 No 3, 2003;
- Stacey, R. (1996). “Complexity and Creativity in Organizations”. ISBN: 978–1881052890;