What Agile Doesn’t Address

Dear Software Industry,

I’m writing to you today to address some gaps which need to be spoken to. In doing so, I don’t want to appear ungrateful — you’ve provided some amazing technology which has transformed society as we knew it before. I’ll let historians of the future debate the goodness of that — I don’t wish to digress. That’s not why I’m here.

I’m here to talk about Agile. And SAFe. And a dozen other sects in the religious war which seems to be raging all around and through you. The sheer weight and fury of articles on these topics is, at times, overwhelming. And, as with most religious fervor, the true goal of what we’re all trying to achieve — the finish line that ought to bind us all together in agreement — is obscured by the maelstrom of flung mud and smoke and mirrors.

What is it we’re all trying to do? I’d phrase it like this:

“We are trying to build maintainable, efficient, cost-effective, defect-free software to address pain points of real customers and deliver them rapidly to an environment where customers can experience their benefits.”

How’s that sound?

Now, I’m going to say something a lot of folks probably think is contraversial. Agile (or Scrum, Kanban, you name it) is probably 5% or less of the solution to achieving that. Unfortunately, it’s the 5% that us left-brained engineer types feel competent addressing.

There are two other much bigger factors to get quality functionality out the door :

• Clarity of the definition of what’s being asked for
• Rapid, effective communication of that ask and any changes to it

Let’s start with the second one first. Agile’s manifesto was a direct response to the FACT that in the real world, requirements change. History is full of very well designed, quality engineered products which, thanks to the length of time it took to get them from on-paper to market, failed miserably because the market changed. Ford’s Edsel is a great example. It’s not hard to find others.

So yes, rapid effective communication is not just critical, it’s paramount. And yet the Manifesto then goes and screws up by saying “Individuals an interactions over processes and tools” Bzzzt. I’m sorry, a white board is a tool. A Miro board is a tool. And, properly utilized, these are far, far more effective a ways to communicate than 30 people all having 15 individual face to face conversations.

Bear with me.

Let’s look at the ways 30 people could share an idea effectively. We could arrange them in a circle and have the first person whisper to the second and go around the room. You might remember Telephone Game from kindergarten. This pattern is doomed to fail then and it’s doomed to fail now. And yet

• Requirements -> Architecture -> Design” -> …
• Epic -> Feature -> Capability-> Story

We are using that pattern each and every day without questioning it and wondering why we fail. There’s a better way.

Let’s put one person in front of the group and have them tell everyone what to do. More efficient, but likely equally ineffective. There’s 28 chances that there’s a better idea out there (I’m discounting Bob, he’s dead weight and everyone knows it). Those 28 chances are not going to be heard.

How about this? Let’s put a whiteboard up there in front of everyone with a set of related use cases. Let’s DRAW a DIAGRAM which, step by step, speaks to the logic and data flow which will need to be taken to achieve that functionality.

“Wait!” I hear Bob saying, “That’s waterfall! It’s too big.” And for once, Bob is right.

It takes a village to create that drawing — it starts with a top-quality product owner or BA interviewing the business (not the customer, they know pain points, not solutions; symptoms, not cures to disease). This interview process literally goes like this:

“Tell me what’s the ask…what triggers the system’s behavior and what is the proper behavior of the system when it gets that trigger.”

Different flavors of trigger (and the underlying data state of the system) will yield your thin vertical slices. But wait, there’s more.

We then arrange that in a diagram:

We then ask “Tell me, in 8 steps or less, how do we get from the green arrow to the red one?”

It’s amazing how well the mind works step by step. We now have a basic “spine” of a solution. The so-called Happy Path.

But wait, there’s more.

Let’s distribute “hats” to the 30 people in the group. These hats speak to their expertise — Joe knows the company’s current and envisioned architecture really well. Hakkim knows Front End stuff. Zarabeth knows business logic and what reusable microservices are there. Tia knows the database. Carl knows federal regulations. You get the picture.

These are not job titles or job descriptions. It’s silly to shoehorn someone into being allowed or not to speak up simply because of how HR categorizes them. People have valuable experience — let them bring it to the table.

Each person, in their own good time (not in a meeting, that’s not how the brain works. Let the muse come to them) will tweak the diagram a bit, adding notes and commands, reworking the arrows.

This input must be timeboxed — we don’t get all year to come up with the perfect solution, we just need something that works and is encapsulated so we can upgrade and flesh out later.

Give ’em 48 hours or so.

Now we drill down.

Our team came up with 5 “sections” of the solution. Here’s where things get really cool!

1. Trigger and Behavior becomes our E2E test case: “GIVEN I’m about to do <trigger>, WITH <data> on the trigger arrow and <data state> embedded in the system, WHEN I trigger, THEN the system delivers behavior”

And if we specify our clauses in that enhanced-Gherkin statement so the evaluations can be performed without a human being, we’ve just made E2E testing automatable. (Why enhanced? Because most test cases differ solely on the data states involved (does the SKU match with something in the database, is the person authorized to log in) with identical triggers and actors. Calling data state out separately is a best practice we’ve uncovered.)

2. This process is recursive. That light blue “WHAT” box up top? Let’s expand that to be its own entire diagram. What’s it’s trigger(s)? What is coming out of those 2 white arrows and under what conditions do we see control and data transfer via one or both?

Divide and conquer here — drill down and DECOUPLE. It’s not about independent teams, it’s about decoupled functionality — limiting the COMMUNICATION that needs to take place between people and components and increasing the RELIABILITY of standardized connections among well defined system elements. This drives maintainability, reusability and cost effectiveness.

But wait, there’s more.

3. Each of those white arrows is a unit test! Yep! By simply diagramming the flow of the functionality, I’m writing unit tests (or, for higher level drawings, integration tests) as I go:

GIVEN I’m at step WHAT, WITH data to be passed on arrow What-Step and <underlying data state of <step>, WHEN I trigger that state transition, THEN I arrive at Step (and how do I prove that?)

An arrow == a gherkin. Comprehensive, 100% code coverage testing — simply by drawing a solution in a well structured, non-English diagram.

And if the eGherkin is rich enough, you can pass it straight to an automation proxy and build test cases without a human being in the loop to screw that up. Tools are a good thing.

But wait, there’s more.

Non-English is critical here. English is one of the most ambiguous grammars on the planet — and for good reason. Human languages have evolved to keep our minds happy, not to be precise — quite the contrary. If we communicated in a regular, parse able, non-ambiguous grammar, we’d go insane just talking to each other at the water cooler. No, our minds need body language, tone of voice, words, context and a whole lot more.

The INSTANT we write something in prose and put it up in front of those 30 people in the room, 29 are GOING to misinterpret it. Guaranteed. And it gets worse if that prose is in a language not native to the reader (or writer).

Software is too expensive to leave to such inefficient and ineffective communication patterns. There’s very good, scientific reason to put an unambiguous diagram in front of the group and let everyone DERIVE the portion of the diagram they need.

Buh-bye telephone game.

But wait, there’s more.

I was walked into a room about 9 years ago during the first week of a new job. It was sprint review and the Devs were on one side of the table with the PO’s on the other (yes, multiple POs, also a real-world norm). They were arguing over whether the software built in the past 2 weeks met the ask.

It’s not important whether it did. It’s important to realize that this scene should never be possible. An unambiguous, non-prose Single Source of Truth artifact like our diagram is half the solution. The other comes from Albert Einstein.

If you can’t explain something, you don’t understand it.

Ok, not exactly what he said, but close enough. A crucial missing ingredient in the above conference room is something which didn’t take place 2 or 3 weeks prior, and that’s having the receiver of a message explain the ask back to the originator. In SDLC speak, this means that yes, we need to take the time for each person (assuming there’s a chain of communication at all, which we’re going to work VERY hard to eliminate) to EXPLAIN THE ASK back to the person doing the asking. If what’s coming out of that recipient’s mouth is gibberish, so will their code be.

This only takes a couple minutes and you’ll feel stupid doing it the first few times, but it serves two very important purposes:

1. Hearing yourself verbalize something locks it in your mind. It’s a psychological thing, and not one the Agile Manifesto folks probably knew about. But it’s real and it’s effective.
2. It’s a structured form of face to face communication, not just a lecture, but a “stand up and explain that back to me”. It forces the recipient to THINK THROUGH what they are being asked to do, and the mind can’t help but poke holes and ask questions.

But wait, there’s more.

Let’s look at the things we can derive from that diagram, shall we?

• The diagram is like a blueprint. And like a blueprint, it’s content is contributed by a team, not authored by an individual. See that arrow up there where financial data is being passed from outside the firewall to inside? SOx compliance, legal and financial needs to look at that. Easy to tell that in a diagram, really tough to drive stakeholder engagement that well through prose.
• As we said, arrows == Gherkin. And the definition of bug-free is “Expected Behavior aligns with Observed Behavior across all code paths”. We can achieve that using this approach — as long as the folks wearing the “test” hats do their job and make sure all code paths are identified and the system can produce evidence of correct behavior across all those paths.
• Those behaviors extend beyond deployment, btw. In fact, the job descriptions, to an extent, of the test engineers and the operations staff HEAVILY overlap — they’re both verifying that expected behavior is manifested, one team verifies that behavior before deployment and the other monitors and verifies it after. It’s all just behavior, though, which means the system needs to be able to produce evidence of that behavior — which means the Devs need to build in that ability to produce that evidence. It can’t be cobbled on after the fact, it has to be baked into the design. Test hats verify that.

So what do User Stories look like in a world such as this? To be honest, they don’t exist.

It’s not the diagram that’s the ask. It’s the change between version N of the diagram and version N+1. A new system starts with a blank sheet of paper, so the entire diagram is new. But after that, we make incremental changes to portions — an arrow here, a step there. Those deltas are our change requests — our artifacts to be worked on and deployed.

Our diagram continues to be the single source of truth. No stories, no story points.

Tasks.

This is where the development team looks at the diagram, step by step and says, “it’s going to take me X, Y and Z to alter this box to manifest this behavior.” Not the whole thing, just that one box 5 levels down on the diagram stack. And this box. And that arrow. One tiny little diagram adjustment at a time.

Even Bob can estimate at that level. And Bob can estimate in real world units, not magic beans which have no units of measure. “That’ll take me 2 hours.” It DOES NOT MATTER if the estimate is right! Because as we estimate we’ll TREND to getting better and because we’re going to collect ACTUALS as we go (in the same real world units of measure), we’re going to be able to generate a multiplier so that Bob (or Tia’s or Hakkim’s) estimate can be adjusted to be more accurate (cause everyone knows Tia always says she can do things in 2 hours when it takes 4, so her multiplier will be 2).

Real world, really accurate estimates from the ground up which can be backed by science. Business Team, if you don’t like the numbers, deal with it. Want it faster? Cut scope — or realize that it’s nice to want. Those are your choices. Because this team needs to be able to sustainably produce functionality for the long term and asking them to make a Herculean effort to get your pet baby out the door isn’t going to happen. Scope, Resources and Timeline is what you have — cut scope or elongate timeline. Resources are fixed and doing a mighty fine job. Don’t upset that apple cart.

Dev team — you can process those tasks using whatever pipeline you want. Kanban works — pull a task off the queue when your WIP count gets small enough. Scrum works if you want that. Doesn’t matter — the quality of the ask is assured, and you explained it back to the PO, so you know what to do and how to do it. We trust you to get your job done. Even Bob.

What does this work pattern look like in practice in the real world? 30+ Scrum teams to date and without exception 100x reduction in bug counts and 2x to 3x increase in delivery of real, new, valuable functionality. Speed + Quality = Possible. There are years of hard metrics in Jira and Rally in several organizations to back this up.

Moreover, this work pattern evolved from literally HUNDREDS of retrospectives. So it can’t help but actually BE Agile in the same way that we Homo Sapiens can’t help but BE from the same family as Australopithecus.

I’d like to think that Agile, which is supposed to retrospect itself to discover better ways of working, would realize this is a good thing and only natural. And Agile would also realize that if metrics consistently prove a work pattern to be better, it’s a crime to not distribute that work pattern to the masses regardless of what some dusty, 20-year-old “right for its time” document touted two decades ago. Times have changed — and this is better. Let’s apply energy productively and help us make it even better still.

And what about those human beings working on the teams? Morale is, frankly, amazing. Nobody wants to leave such a team because the delivery is predictable, stellar and guaranteed. Nobody gets beeped at 2am with a Sev-1 prod issue because it’s literally not possible (other than race conditions, but this is already a long article, I’ll speak to those later). So, people can go home, be with their loved ones and forget about work and enjoy being on this planet — because at the end of the day, THAT is what matters.

Contact me if you want to know more or want help in moving in this direction. I can help.

Software Industry, thank you for your time and congratulations if you’ve read this far. It’ll be interesting to see if anyone actually gets this far down this tome. If they do, I’ll ask them, “And what was the magic word embedded in the middle of the article?” Let’s see what people come up with.

Best Regards,

Steve