Change is Software’s greatest threat, and its defining characteristic. Disagree? How’s that Agile Transformation going for you?
To explore this cognitively-dissonant statement, we’ll chase some waterfalls as we look at what agility might mean to Software Engineering, and the role that “change” plays in it.
In the “old days” Software was built with months of requirements gathering and several revisions of comprehensive specifications across various disciplines.
Back when I worked at a prominent ISP, our Product Management group would spend months drafting comprehensive requirements for a new feature or product, into a glorious Product Requirements Document (PRD).
After 7 years climbing my way up thru seniority ranks with a strong track-record on smaller-scale projects, I finally got to lead my first big cross-functional project and earned the honor of producing a no-less glorious Technical Architecture Document (TAD). Rife with Architecture, Entity-Relationship & Data-Flow Diagrams, it also covered:
- Database Schemas
- System load estimations, load-testing & caching strategies.
- Provisional release engineering procedures, to define how we would deploy code to Staging and Production.
- Logging requirements for application health monitoring
- Business Metrics tracking
The QA team would concurrently pour over the PRD to produce a comprehensive test plan.
Our Systems Engineering team would analyze my TAD, estimate hardware requirements, and proceed to plan out its provisioning and deployment.
Finally, we’d start coding, over several months. Then came weeks of QA … and bug fixes.
There could only be two possible outcomes:
- We launched the project as fully-specified
- We failed
This all was known as the Waterfall Model.
How could we fail with all this planning? Well, as you might have guessed, deadlines are “a thing”. So failure to launch by a given deadline is, by and large, a failure. Of course we’d do our best to plan for this, and it went a little bit like this:
- We need to launch by July 1st.
- To meet that deadline, QA will need to start testing no-later than June 1st.
- According to my TAD analysis, development needs to start no later than March 1st. Hopefully earlier. Preferably yesterday.
- So work on my TAD needs to be completed in February, and it should in theory also give Systems Engineering enough time to have the hardware commissioned for QA by June 1st, and Production by mid-June, hopefully earlier.
- So the PRD has to be finalized in January, hopefully having given me enough time to ask for revisions for clarifications. In some instances, I was able to negotiate reducing scope, based on early-eyeballing of the headaches that await us.
Of course you might read this and ask yourself: Did it even make sense to define a launch date of July 1st, ahead of having fully-completed the TAD? Don’t you need that to estimate development efforts, which will impact Systems Engineering efforts and delivery to QA, which will impact the launch date?
Well … We all know what July 1st represents: It’s the “End of Q2”.
With all this said, against many odds, we delivered. Right on schedule. And the product generated significant revenue. By all measures, it was a success. A large part of which was driven by the platforms we’d built, the technical debt we had paid-off, the team maturity we’d achieved, and the skill and dedication of my wonderful peers.
What were we trying to achieve with this process?
Looking back on all this, in my opinion, this entire process existed to mitigate one thing, and one thing only:
With so many cross-functional moving parts, fixed deadlines and all-encompassing product scope, the most dreadful thing that could happen to all of us operating at the bottom of that proverbial hill, were changes to what the product should do.
So we did whatever we could to “lock-down scope”. “Lock-down requirements”.
Completeness was a thing: Scope was complete. Design was complete. Code was complete. Code was even, at times … frozen.
Change was a fierce and feared predator, and much procedural friction had to be put around it to contain it.
In light of these symptoms, I would argue that “Change” was already viewed as Software’s Greatest Threat.
At the time, being an ISP, users of our products were by and large a “captive audience”. We could afford to fight change. How realistic do you think this would be in today’s World, or for most emerging companies of those days?
How long could we realistically keep change in its cage?
While the value-proposition did land well on my project, many projects weren’t so fortunate. The last thing anyone wants is to spend 6 months to a year engineering a project, only to realize from user feedback that we need to … “Pivot”.
That’s right around the time you run out of Series A money.
In recognition of these challenges, thought-leaders in Software Engineering got together in Utah and drafted the Agile Manifesto.
If I were to boil-down the “Essence” of the Agile Manifesto to accommodating one thing, it would be …
Embrace change: Ship smaller things early, often, get feedback from end-users, course-correct along the way, when it’s cheaper. Generate revenue earlier.
The Makings of a Manifesto
The manifesto itself is intentionally broad in its definition of principles. It does not advocate any specific methodology or practice.
With this said, many of its authors had for some time been practicing several methodologies that could be classified in two buckets:
- Process & Organizational methodologies such as Scrum & Kanban
- Engineering methodologies such as Extreme Programming
Extreme Programming’s stated intent is to “ improve software quality and responsiveness to changing customer requirements”.
And from Scrum’s Wikipedia article:
A key principle of Scrum is the dual recognition that customers will change their minds about what they want or need (often called requirements volatility) and that there will be unpredictable challenges — for which a predictive or planned approach is not suited.
As you can see, embracing change is a cornerstone intent for both Scrum and Extreme Programming. Combined, they represent a holistic approach to Software Development from the combined standpoints of “Process & Organization” and “Engineering”.
It was at that time that “Change” was recognized as Software’s Defining Characteristic.
When things went wrong …
Having caught a whiff of this newfangled “Scrum thing”, project management ecosystems flocked to it, and were quick to offer training & certification programs for it, packaged into a shiny “Agile Transformation” roadmap, with great promises of highly-functional teams delivering market-leading products.
Enterprises bought into them in a big way, thinking that adopting Scrum would relieve all of their woes with magic unicorn droppings.
And yet, to this day most companies fail to ship software at a sustained, predictable pace.
Engineering fell by the wayside.
Enterprises did what they do best, which is to focus on “Process & Organization”. Executive teams can easily wrap their head around these concepts. But they completely failed to take a critical look at how they were “Engineering” software.
So they kept writing software that was difficult to change.
Scrum sprints would put pressure on software engineers to commit to bold scope, with zero time for design & planning. They’d spin-up some MVC framework, hack-together a bunch of code rife with duplication and tight couplings, and double-down on those mistakes over the course of following sprints.
With the rapid buildup of monumental amounts of Technical Debt, it becomes exponentially more difficult to change a system without breaking things.
To explore these pains in greater depth, I recommend two articles:
Facing these challenges, “Change” continues to be Software’s Greatest Threat.
Getting it Right
Over the course of my management career, I have several times failed to sufficiently improve software that was difficult to change. I could point to numerous excuses, none of which would matter if the end-result comes to failing to achieve certain business goals. And in the end, the buck stops with me.
If I could impart the most significant lesson learned from many battle scars, it would be that now more than ever, it is critically-important for agile teams to have a candid conversation as to what it means to engineer agility into their software, such that it too, might embrace change.
Until we do this, our so-called “sprints” will continue to transform into “marathons” of failed execution.
How might one go about this? How could we write software that’s “easy to change”?
I alluded to Extreme Programming (XP) earlier. Authors of the Agile Manifesto have embraced many of its practices for good reasons. Over many years I too have come to embrace some of XP’s principles, as well as many others, the most impactful of which have so-far been:
It’s always a work in progress. But my goal continues to be to deliver business value at a sustained pace.
After all, should this not be the true measure of Agility?
Exploring what might be the right questions to ask ourselves, should be the first step to addressing the challenges we face.
In follow-up articles, I’ll be diving in greater details into how I’ve applied the aforementioned concepts.