How to prove this new “Agile” thing you’re trying is really working.
Some time ago your organization decided to try out Scrum and now, like any good manager, your boss wants proof it’s working. What will you do, intrepid ScrumMaster?
Your beloved metrics won’t work.
“My team of 5 is doing 40 story points every two weeks do you understand what this means?!”
Your boss may not know a story point from a gummy bear. Even those of us who play planning poker with our buddies on days off know that one team’s 40 points is another team’s 10. And trust me on this one: showing off your burn-down chart is like showing people pictures of your baby — you’re proud and they’re happy for you, but the truth is nobody is as impressed as you are.
The numbers mean nothing outside of your team. But your improvement does.
Track velocity over time. Be able to report an X% increase in work accomplished.
You are not a team of 5 doing 40 story points per sprint. You’re a team that was doing 20 story points 4 sprints ago, has steadily improved productivity since then, and are now doing 100% more work per sprint than you were 8 weeks ago. 100% improvement in productivity makes sense to anyone.
Even better, speak in terms of money whenever you can. Describe productivity improvement in terms of dollars-per-story-point.
If your team costs $500 a week, that’s $1000 per 2-week sprint. Knowing this, you can say, “Boss, our team measures work in terms of ‘points.’ Eight weeks ago, each point of functionality cost you $50 to create. Today a point only costs you $25. We’ve doubled productivity with Scrum in two months.”
I just wrote that and I already want to give you a raise.
The measure of success for any Agile team is WORKING SOFTWARE.
If you can’t show the release of working software, resulting in a real or at least predicted ROI, you have to question why. Our goal is not to do a process, after all. The goal is to make money, get return on our investment. Trying to justify a process without being able to show that it’s doing that is just dishonest. Scrum is about staring problems in the face and fixing them. Dishonesty is an impediment; remove it.
First of all, if possible, show that you’ve released ahead of an original schedule.
You can do this by setting an original schedule based on the early velocity you expect to improve over time. You will keep changing that schedule as scope and velocity shift, but hang on to that original release date, and when you do your first release, compare them. Time is money, so when you release early you’re most likely under budget as a result.
Next, set the stage for reporting real return on investment by having your PO be open about where priority comes from. When grooming a backlog, discussion around priority can easily include an extra sentence about what kind of ROI is driving that priority.
Again, try to put dollar values on it. Clickthrough rate increases, conversion rate increases, app download increases, bug report decreases, product review score increases. What’s the reason we’re doing these product backlog items first?
Whatever the ROI targets of a release are, know them in advance. Then, after release, verify that you got the return if possible. If it’s too early to verify, brag about why you know it’s coming.
Beyond ROI, working software after each iteration has it’s own value as technical inventory. Measure how many iterations have resulted in working software that is ready to ship even if it wasn’t released. Come up with creative other uses for the working software even if the product/project was abandoned. Explain the benefit:
“Even if you abandon the project today, there’s working code here we can ship or use elsewhere. That’s work you won’t have to pay for later. In a waterfall project, you’d have nothing but sunk cost at this point until the project is finished.”
Happiness matters… right?
A wizard appears in a puff of smoke and booms, “ScrumMaster! With a twitch of my aged outstretched hand I will give you wealth beyond the grasp of your imagination… but you will never experience happiness again. What say you?!” No one would take that offer and we’d all walk away really happy to have seen a real life wizard.
(Alright, alright, so you would accept. You’d become a martyr, sacrifice all of life’s happiness and spend the newfound riches solving all the globe’s problems. The wizard rolls his eyes. Not the point.)
We all want to be happy and life is by definition pretty bad without it, so yes, happiness matters. You’ve spent your whole life pursuing it, in fact, so it might be the only thing that matters.
Nevertheless, many managers will not be very impressed by a graph showing a trend of steadily increasing employee happiness since the introduction of Scrum. Remember, we need to speak in terms of money whenever possible.
If you’re going to sell the happiness metric, your case must be: “increased happiness equals lower turnover, & lower turnover equals more money.”
Less resources spent securing and training new talent is less spending. Furthermore, the time spent looking for and training newbies is instead invested in removing impediments of current employees and that steady improvement in productivity we talked about above.
So when asking your teams about their happiness, the real question you want answered is, “How likely are you to quit in the next year?” But phrase it that way and you’ll never get an honest answer. See if you can get a regular, measurable answer to a question like, “How much are you enjoying working on this team?” Then you can hopefully share an improvement in morale on your Scrum teams, and explain it’s meaning: Scrum is protecting the investment in your workers.
By the way, while you’re gathering this info, be sure to ask team mates what can be done to improve the score, then make it your kaizen, ScrumMaster.
Better than waterfall? Prove it.
“So,” your boss says, “you’ve had successes. How do I know these successes wouldn’t have happened in a waterfall project?”
That’s tough. No matter what, unless your company executed the same project simultaneously with two teams, one Scrum, one waterfall, the comparison with another project will seem like apples to oranges. Sure, you released twice as fast; the other project had bigger scope, had fewer people, had more scope changes…
The best suggestion I can offer: gather the same metrics on ALL waterfall projects and ALL Scrum teams. Then calculate averages and compare.
Time to release, improvement in productivity, happiness, total cost; whatever the metric, compare the average Scrum team’s score to the average waterfall project if you can. Ideal, but not always possible.
Another option: find instances where something happened that you know leveraged one of Agile’s strengths. And the main strength of Agility: dealing with inevitable change.
Find specific instances when requirements changed at the same time they would have changed during a waterfall project’s post-requirements phases.
This is usually a late change due to an external influence, not a change resulting from clarification of requirements that were low in the backlog priority (a waterfall project would have clarified such a “change” before beginning any work).
For example, did a competitor launch a product you had to scramble to match? Your Scrum team probably responded to that high-priority change within the span of a sprint or two. A waterfall project would have gone back to the drawing board or had a much harder time switching developer contexts, both costing time and money. This is a story you can tell to compare your Scrum team’s performance to a hypothetical waterfall project. Just be sure your hypothetical situation is one that nearly everyone can agree would have happened.
Improved productivity, inventory of working software, quicker ROI than waterfall and with a faster response to change, all spoken in the universal language of money: a few ways to measure for your dubious boss — and yourself — whether Scrum is working. Best of luck.
If you feel like I’ve missed something, please contribute! Leaving a note or catch me on Twitter. Thanks for reading.