The metaphors we use to try to understand how software projects work have a profound poverty. The way many organizations approach software projects still mirrors how we approach a construction project: we describe what we want, we make a design, we create a plan and a timeline, and then we begin work based on the plan. The vast bulk of the plan is expected to be known and understood before work begins. This is completely appropriate for a construction project, where the design decisions follow well-known patterns. However, it is inappropriate, and even counter-productive, to assume that we can make a comprehensive plan at the beginning of a software project.
That being said, we still make contingencies when building houses. Construction projects’ plans aren’t 100% comprehensive. While construction is ongoing, we still have the opportunity to waffle about paint colors. We can adjust the layout of the kitchen. We can add a second sink to the upstairs bathroom. However, house-building is still a solved problem. The process of building a house is well-known, well-understood, and is a well-trodden path. Despite the contingencies and last-minute adjustments, accepting and responding to changes in the plans of building a home fall in the margins, because the most challenging decisions have already been made. The mid-project changes, in the grand scheme, don’t matter much. However, treating course corrections in a software project as marginal is a big mistake for two very important reasons.
First, software is just so complex. The depth of sophistication in a typical software project dwarfs the design of even the most intricate and stately of homes. The interacting systems that work together to make a home function properly are so well-understood, isolated, and tangible that it’s almost ridiculous to think of it in terms of a “systems integration”. In comparison, the complexity of software and the various systems that it interacts with is staggering. Nobody marvels that a house has both electricity and indoor plumbing. It’s not particularly noteworthy if a home has both satellite and cable television. There are no worries about vendor lock-in if you choose a particular brand of window or insulation. It’s incredibly unlikely that your light sockets can have a critical security bug that will unlock your doors for an hour every night.
Second, the raison d’etre of a construction project is blindingly obvious. Rarely does one ask, when seeing the construction of a house, or an office building, or a shopping mall, or a skyscraper, “What’s that thing for?” The purpose — the value it provides, the utility of it — is well-understood. Not so for software projects. In fact, many software projects exist because someone wants software that does something that hasn’t been done before. Its purpose isn’t a well-trodden path. In fact, the novelty of a software project is its primary reason for being. Whether it’s to add efficiency to a business process, modernize antiquated workflow, entertain, communicate, convey information, or any other purpose, a software project is begun because someone with a checkbook decided that the world needed software that doesn’t yet exist. Despite how routine we want to believe they are, software projects are always blazing trails: you’re building something that nobody has ever built before. If there’s already a piece of software out there that does the job, why bother building another one?
These two things — complexity and novelty of purpose — suggest that we should approach a software project differently than we should a construction project. Let’s consider an example. Given the traditional model, we have someone that says, “I need some software that does ______.” This desire is refined until there is some statement of requirements, usually a document. This requirements document describes how the software should function. The size of the document and the perceived complexity of it gets translated into some kind of size estimate. This is the project’s scope. The ideal scenario is that given some known quantity of scope, you can calculate the project’s length in days, weeks, months, or years. The ideal project’s timeline looks something like this:
There’s an important element here that is ignored: because of the two ideas we discussed above — complexity and novelty of purpose — scope isn’t a known quantity at the beginning of a project. We think it’s known. We believe that we have a good high-level estimate. However, what inevitably happens is that scope grows. As you begin work on the project, the details come into view, the assumptions are challenged, the incompatible requirements are pitted against one another, unexpected complexity is revealed, and the scope grows. The industry even has a term for this: scope creep. When scope grows, we call it scope creep, and it’s generally viewed as A Bad Thing™. Therefore, everyone involved in the project is expected to have their scope creep filters on, discarding any scope that doesn’t pertain to what the original (and often contractual) estimates dictate. Despite this, omissions, misunderstandings, and new information always exert a force on a project, causing its scope to grow. This is a natural progression of software projects: it will happen. If you persist in managing a software project like building a house, the project, over its lifetime, will often look more like this:
A moment occurs when the growing pile of scope that can’t possibly get finished by the deadline suddenly disappears. This event is when the realization occurs, late in the project, that we can’t possibly meet the deadline at the current pace. In response, scope gets ruthlessly trimmed in large swaths, and the project still careens beyond the deadline before it’s “done”. Even though we trim scope, the breadth and depth of the scope changes cause more disruption than the project can absorb in such a short time.
Keep the Trimmings
Let’s add a little more honesty to the prior graph. This graph is the same, except we’re also illustrating the discarded scope; the things that everyone either ignored or withheld stating, “that’s out of scope.” Whether that’s a lone programmer at his desk deciding to invest minimally in this feature or that, whether it’s a decision a group reached in a meeting, or it’s a requirement that was intentionally ignored, valid scope (i.e. work that is valuable) gets discarded constantly throughout the project.
Notice how the new band doesn’t get any smaller at the scope trimming event? Those features that we trimmed didn’t just disappear! They just got moved into the “out of scope” bucket. Despite how a contract is structured, or a project is managed, there will always be more to do than there is time to do it. The scope at the outset of the project is never the same scope as when you look back on the project in retrospect. We can continue to view this through a negative lens using disparaging terminology like “scope creep,” or we can accept it as it is: a fundamental force of software projects. Let’s frame the rest of this discussion around these two axioms:
- Scope and time have a roughly proportional relationship. Increased scope requires more time to build. Reduced scope decreases time to build.
- Scope always increases.
The pathological case of these two axioms is all too common: a never-ending project where scope grows faster than it can be built. But it doesn’t have to be that way. In fact, the fear of scope creep and the resistance to it is precisely the reason why project timelines get blown.
We must not fear scope creep. Scope will grow, and that’s okay.
The most important realization in managing a software project is that the understanding of the project’s scope at the beginning of the project is almost always inaccurate, and usually, wildly so. Therefore, the timelines, architectures, and milestones derived from that initial, inaccurate guess at the scope will only magnify its inaccuracy if they aren’t constantly adjusted throughout the project. As a good friend of mine often reminds me:
There is no such thing as scope creep, there is only deeper understanding.
— Jason Knight
Let’s also examine the assumptions of why someone brings a budget to a software development team and says, “build this for me.” In general, this client believes that the scope they understand at the beginning of the project is the minimum viable product (MVP) and that when the project is “done” they can begin obtaining value from the software that has been built. In other words, the client gets no return on investment (ROI) until the project is “done”. So, naturally, the terms scope and MVP appear to be synonymous at the beginning of the project.
Let’s consider a project’s timeline a little differently. What if we frequently re-evaluated what the most important, critical-path features were, built those first, and also continually re-evaluate the distance between the currently-built software and our current understanding of MVP? Perhaps our project timeline might look a little more like this:
Notice how the green line, indicating what the total scope of the MVP is, descends, even though the total scope, overall, increases. Rather than fearing and resisting “scope creep,” we take advantage of the deeper understanding that we gain at each step. When we continually re-evaluate the software’s purpose, we can find opportunities to shorten the path from what has been built to software that delivers value. We should frequently evaluate the work that we’re doing so that we’re always working on the most important thing.
Are We There Yet?
Do we call the project “done” when we’ve delivered the MVP? Well, that, of course, depends on the environment surrounding the project. However what is certainly true is that when the MVP is delivered, there is still plenty of work left to do. If we call a project “done” when there is no work left to do, then a project will often never be “done”.
This leads us to another fundamental truth of software projects: software is never “done”. Earlier I stated that two elements of software projects — complexity and novelty of purpose — require us to think of software projects differently. If software can’t ever be “done” then we must view a software project not as the construction of a single artifact (i.e. a software “release” or “version”), but as a stream of small artifacts, each of which is potentially shippable. When we deliver software in small increments like this, naturally, we ship the critical-path MVP features first, leaving the low-value and low-risk features until later.
When we view the construction of software in this way, deadlines become far less scary. A deadline is simply a cutoff point when we say that the software is “done enough”.
If we always deliver the most important features first, features that are at risk of slipping beyond the deadline are, by definition, low importance.
TL; DR
Rather than continuing to corrupt our understanding of software development by misusing the construction project metaphor, let’s consider that we might still need to give consideration to a fifteen-year-old document that speaks to this very issue:
Customer collaboration over contract negotiation
Responding to change over following a plan—excerpt, http://agilemanifesto.org
Rather than relegating unexpected changes in scope to the margins, perhaps we should consider that change is not only expected, but welcome. We can deliver value much faster if we do these three things:
- Deliver software continuously.
- Embrace new insights instead of rejecting them as “scope creep”.
- Continually re-evaluate what the MVP really is.
Dealing with change deserves first-class treatment in how we build contracts and establish expectations with our stakeholders. If we, as software professionals, can respond to change in a way that still fast-tracks the project towards ROI, we’re in a really good place.
