Agile Is Not Enough

Truly successful software delivery requires far more.

I want to state for the record that I am not anti-Agile. Some of y’all may have assumed that based on the title and I want to assure you that’s not the case. I use Agile as frequently and as comfortably as I use a Phillips head screwdriver. And I use it the same way — as a tool, one tool in a collection of other tools.

Because Agile, alone, isn’t enough to deliver software effectively.

Who am I to say this? I’ve been “doing” software for over 40 years. Developer, Team Lead, Sr. Engineer, Product Owner, Product and Project Manager, Scrum Master …I’ve pretty much done it all.

Over the past dozen or more years, I’ve come to embrace Agile and quite a few of its frameworks and added them to that figurative toolkit.

But I’ve done more than that. Across those years and through many retrospectives, I’ve come to realize that these tools are fractional solutions at best. And there’s real danger in believing that they can do more than they’re capable of.

Let’s further this DIY/Home Handyman analogy, shall we?

To me, the industry’s current hyperfocus and religious wars on Agile and Scrum’s strengths and weaknesses are like arguing over what weed killer to use on that patch of dandelions in the back yard when we ought to be focusing on the flowerbeds out front — and in fact, overall curb appeal of the house.

Curb appeal is all about the front yard, not the back. And successful software delivery is about producing working code (lawn + flowerbeds) so we can positively change the behavior and lives of our customers (curb appeal).

In other words, success is getting the most correct software out the door, based on current understanding of all customer pain points. Moreover, we want to do that as rapidly as possible while maintaining the ability to pivot should our observed behavior of the codes effects on customer behavior not align with our expectations.

Follow the logic here: if we can determine that this code did or did not change our customers’ lives for the better, that implies that we instrumented our delivery in such a way that we can detect that “betterness” (or lack of it) — which also means we sat down and really thought through what “better” meant in this situation. And we had baked the ability to collect those metrics into the system design!

Industry can’t make up its mind what to call this, Behavioral Driven Design vs Behavioral Driven Development, so I’ll just call it both: B3D. Because expected behavior of a system is paramount to getting it right. And that right-getting starts at the Customer Journey level. It’s a must-have addition to Agile/Scrum.

B3D. Awesome tool to add to our kit. But wait, there’s more.

Back in kindergarten, we all played a game where we sat in a circle. Some chosen person, usually the teacher, whispered to the person to their left. That person, in turn, turned and whispered what they thought they heard to the person to their left and so on until the “message” made its way back to the initial person.

We all got a huge laugh when “World Peace” turned into “Rice Pudding” — but then we moved on past kindergarten and something horrible happened. We forgot that lesson.

The Agile Manifesto proclaims that “Face-to-face conversation is the best form of communication”. In Telephone Game, each and every one of those conversations is, in fact, face-to-face. Could the almighty Manifesto have missed something? Well, yeah, it misses quite a lot of things.

Quality of communication is a big one.

Over time, the software industry has enshrined the doomed-to-failure Telephone Game communication model and leveraged it to be the backbone of our SDLC processes.

“Requirements Document”, “Architecture Document”, “Development Notes”, “Testing Document”, “Epic”, “Capability”, “Feature”, “Story”, “Delivery…”. These are just fancy terms for those individual kids sitting in that circle.

We’ve forgotten the basic lessons we learned in kindergarten. Shame on us.

The 100% failure rate of this communication model is a strong hint why a great number of software development teams produce bugs. And the best Phillips head screwdriver in the world won’t stop that from happening.

Let’s talk about bugs. After all, a lack of bugs is fundamental to the notion of “working software”, and the Manifesto is spot-on with its focus on working software. Unfortunately, it doesn’t really define “working”. Oops.

Consider the following code:

int result = 10/ 0.0;

We have a problem here, don’t we? Divide by zero! Exception! This code doesn’t work!

Or does it? What if the purpose — the expected behavior — of that code was to demonstrate a divide by zero exception? Within that frame of reference, this code is working perfectly.

“Working”, it seems, depends on expectations. Given that, let’s pick a more precise definition of “Working Software” that I think we might all agree on:

Working Software is when expected behavior aligns with observed behavior. Which means a bug is when misalignment occurs.

Pretty straightforward, right? But let’s look at what this implies.

First, you have to state what that expectation is — and do it in a way that eliminates subjectivity. One of the most frustrating situations in software, and it happens all the time, is when Sprint Acceptance rolls around and the Product Owner and Devs get into an argument whether the delivered code is what was asked for.

Who’s to blame here? Everyone.

• The Product Owner didn’t do their job and specify expectations adequately or properly.
• The Delivery Team didn’t do their job and push back on a change request which hadn’t reached the minimum DoR quality bar.
• The Scrum Master didn’t do their job because they didn’t identify this obvious impediment to the team’s overall throughput.

The fix? Ignore the Manifesto’s silliness around “self-directing teams” and mandate some obvious better practices for change request Definition of Ready: Y’all gotta have a statement of expected behavior for whatever changes you’re making.

Now, about that subjectivity thing, why’s that bad? Because automation is going to have to verify alignment between expected behavior and observed, and a dumb testing script can’t make a value judgement.

Great, we have wonderfully phrased, unambiguous statements of expected behavior. Now what?

We need equally wonderful guarantees that we can observe that behavior when the time comes.

Too often, I see Telephone Game based SDLCs which engage “The Testing Team” after the code’s been written, and too often, I see that Testing Team throw up their hands in frustration because the coders haven’t bothered to expose evidence of expected behaviors. The code isn’t testable. And inevitably, the coders (and I’m pointedly NOT using the term Software Engineers here — it’s not applicable in this case) are busy off writing their next big block of spaghetti. So untested code paths get pushed to production and the resulting system misbehaves.

All the face-to-face conversations between these teams won’t fix the upstream problem: The change request didn’t mandate exposing evidence to observe the expected behaviors.

Inevitably, those “too busy to fix it” devs won’t have enough time now to make things right — but they’re sure going to have to find time (and capacity) down the road to do it over.

Speed + Quality == Possible

In October of 2019, a small group of teams within a Fortune 100 company cast off their contracted “managed service partner” and embarked on a journey to bring development in-house. Management was awesome — servant leadership personified — but the team had to show results.

The challenge was formidable. The production system was a plate of supply-chain spaghetti: various flavors of SAP/ABAP, Java and some one-off custom technologies steeped in mysticism and tribal knowledge.

But the team had Agile. And the ability to think outside the box.

They self-imposed some policies and procedures that Agile purists might cringe at:

• Change requests (aka Epics, Features, Stories, whatever) must always have (1) A customer voice (the ask, or the “starting line”), (2) Gherkin-based acceptance criteria (the finish line) and (3) Concrete, well thought out and peer-verified tasks to get from (1) to (2), estimated in real hours, not magic beans, by the person who was going to do the work.
• These asks must be represented by diagrams with very specific structures and rules about how to craft them. Thankfully, these rules and structures took about 5 minutes to learn.
• With the “happy path” as the backbone of this Single Source of Truth (SSoT) diagram, stakeholders added additional data such as architectural alignment, tasking and estimation, testing and operational needs, validation by external stakeholders such as performance, accessibility, SOX compliance, etc, and so forth.
• DoR was defined as “The diagram has been reviewed by everyone with a behavioral stake in the resulting system and no further changes are forthcoming.”
• Each downstream segment of the SDLC then derived what they needed as a filtered view of the SSoT. Agile still happened — if late stage changes were required, as they often were, the diagram was changed and teams re-derived the subset of information that met their needs.

The result?

• Stories massively shrank in size. A 3-day story became a rarity and 2-hour stories not uncommon.
• Velocity of delivery initially lagged as Product Owners habituated and refined the new behaviors. It then zoomed far beyond expectations.
• Bug counts dropped almost immediately as design-time engagement of testing and operations identified all code paths and ensured quality statements of expected behavior and collection of corresponding observations.

Management smiled.

Then the teams noticed something they didn’t expect.

Morale soared. Turnover dropped to near zero. Which boosted throughput even further. And six teams of a dozen members each began a nearly two year run of flawless, bug-free deliveries.

In doing so, they appeared to violate three of the four Agile values:

• They used tools to ensure alignment with what their retrospectives told them were best practices. This drove adoption because the best practices were the easiest follow. Team members would have to do extra work to do things wrong.
• The Single Source of Truth diagrams were, quite frankly, comprehensive documentation. Certainly not the tomes of prose the Agile Manifesto gurus were speaking of, but comprehensive documentation nonetheless. Comprehensive and invaluable because they placed a single, unambiguous non-English artifact in front of the class, eliminating any chance of Telephone Game and forming the basis of laser-focused face to face conversations where those were needed. By making misinterpretation impossible, it also largely solved communications with ESL off-shore teams.
• This methodology was definitely a plan that the team followed, but it wasn’t a plan of what to build. It was a plan of how to build software in general: how to ensure that self-imposed artifact standards were met. This addressed Agile and Scrums silence when speaking to quality of what moves through their efficient pipelines.

Another gap bites the dust.

At the same time, these teams fully embraced the value of Customer Collaboration by engaging all stakeholders (folks who needed the system to behave a certain way, internal and external) early and predictably so those behavioral expectations got designed in from the start.

Agile, Alone, Isn’t Enough.

Scrum, Alone, Isn’t Enough.

These teams’ experience isn’t an isolated incident. Dozens of teams have embraced B3D and Microstories and, without exception, have achieved similar results.

Flawless, rapid delivery of customer value is possible — repeatably and predictably — if we stop focusing on the dandelions in the back yard and start looking at what is truly keeping us from what we’re all trying to achieve.

#agile #BDD #ATDD #scrum #telephonegame #SDLCQuality #newagileparadigm #agileisntenough