Forget ROI, let’s talk about Return On Learning
In one of my recent classes, we examined how you could combine Lean UX with Scrum. It’s a kind of an olive branch training. For years we have had the mantra: “If you can’t fit it in the Sprint, it’s just because you haven’t understood Scrum properly.” But the answer is more subtle than that. But if learning becomes a first class citizen, how do we prioritize “learning” work compared to “delivery of done increments?”
What are you talking about?
Lean UX emphasis learning, and Scrum can provide a great framework to do so. See my previous blog on how to marry Scrum and Lean UX. But now new problems have arisen. Where Scrum uses an incremental approach to deliver value, by building and releasing an increment, will Lean UX supply with simple tools to figure out if something is worth building at all.
To avoid having the discovery (Lean UX team) “run a sprint ahead” of the delivery (Scrum Team) we are exploring how we can integrate discovery and delivery in a single iteration. An thus the question arises how we can order discovery in relation to delivery.
“Whatever you do, make sure you have only one backlog” — Garry Perdetti
What does learning actually mean?
To me learning or, the acquisition of knowledge, is the opposite of risk. You acquire knowledge so you reduce risk. In the context of Lean UX risk however means something else than classical risk management as we were taught, back in the days. Perhaps the best way to explain it is to quote Donald Rumsfeld:
“Reports that say that something hasn’t happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know.”
“But there are also unknown unknowns– the ones we don’t know we don’t know. And if one looks throughout the history of our country and other free countries, it is the latter category that tend to be the difficult ones.”
The green stuff are things Scrum can easily take care of and the blue ones, Lean UX will help us to design an experiment for, the grey stuff is no problem but the orange is the tricky part.
But wait Chris! “What is more important? Green or blue stuff? Tell us! Don’t keep us in suspense!” Patience, grasshopper, all will be revealed in time I first want to drill a little deeper.
How should we look at learning?
Before we compare the value of learning to the value of delivery, let’s take a moment that not all learnings have equal value. So how do they relate to one another. Well, great technique is one that I learned from Laurens Bonnema and is called the risk radar
The missing dimension is the proximity of the risk, e.g. is it likely to be due in a few days, hours or months? You can use a third axis or a separate diagram. What you are looking for is the impact vs the probably.
Another way to look at it, is modeling the known unknowns using a systematic approach. By looking at possible gains/cost/consequences we can use Monte Carlo simulation to generate and review all possible outcomes and the probabilities of any given action upon an unknown situation. A decision tree analysis can then be used to determine the path and choices available. The cost is tied to a percentage of likeliness of an event from happening.
Building models however, comes with a catch:
The Incompressibility Principle
”There is no accurate (or rather, perfect) representation of the system which is simpler than the system itself. In building representations of open systems, we are forced to leave things out, and since the effects of these omissions are nonlinear, we cannot predict their magnitude.” — Cilliers, Paul. “Knowing Complex Systems” Richardson K.A. Managing Organizational Complexity: Philosophy, Theory and Application.
Two cool ways to come up with known-unknowns in a unconventional way are:
1.) Wargame failures
The mind tends to blind for things we are not explicitly looking for, so at one client we spent a week running a “destroy our business” simulation. Locking a group of 4 talented people in a “garage” to simulate a startup that would be entering our domain. Within 2 days people started referring to their colleagues in terms of “them” and “us”. And by specifically looking for underserved needs we found a whole list of potential angles.
2.) Test implicit assumptions
Another way to trigger the mind uncovering what we subconsciously know is starting to make lists of things that “off-course” valid. Chances are that with a simple collaborative brainwriting exercise someone will go like: “Wait a minute! That thing is not a given.”
Can we combine this stuff with other work?
In my book “The Product Samurai” I talk about using multiple factors to drive your prioritization. Typical examples being the usability, purchasability, time to market etc. paired with evidence that you are on the right track.
So classical value buckets as explained by Jenz Broetzmann:
But now they ought to be paired with evidence. A simple, crude way to do this, is simply multiplying the value of each item with the amount of users that support it.
For example: “we believe we can generate a medium improvement to the customer satisfaction and increase our turnover a little by redesigning our website, as supported by the results of our experiment with 25 users.”
So we are getting there, learning drives the ordering of the backlog by supporting evidence that avoids us building stuff we don’t need.
Return On Learning
Of course Return On Investment is still a valuable parameter. As Product Owners we are responsible for the value delivered by the product and the team. But there is a good case to made for measuring learning as well. I see many teams embracing Lean UX principles and increasing their rate of experimentation, which is a good thing! I also see some teams stuck in the learning phase, spending all of their time experimenting and learning and not delivering anything of value anymore. That would be a bad thing.
Return on learning goes back to the unknown/known quadrant above. It means we get better in moving stuff into the known/known area and can take more cautious risks with the stuff we know we don’t know, e.g. using risk radar techniques. It also should improve the cost of learning, e.g. if it costs less to build the feature than to test it, just build it and test it from there!
By constantly improving the speed in which we learn we may improve our return on learning.
Unknown unknowns
It’s hard to spot things you don’t know that are out there, but I have seen two techniques that work:
1.) The way of the astronomer
I did my thesis at the radio observatory of ASTRON in the Netherlands, an amazing place. One of the things I picked up is how astronomers discover new comets. The simple explanation is that you take two pictures of the same piece of sky and subtract them. Anything that remains is apparently moving and new. The math is a bit more complex, but you can basically do the same:
Create a blueprint from the current behavior of your system, then frequently inspect the actual state against this blueprint. If you find changes, you have something that you can investigate further.
2.) It is hard to see with someone else’s eyes
Unknowns are always depended on the observer. There is a famous story about black swans. The people that had never seen them, simply assumed all swans are white. But someone already knew black ones existed. Many teams are composed of likeminded people, so when a vacancy opens on your product team it may pay of to put someone on the team that seems a bit out of place: someone with a background in biology, psychology, neurology, education?
Getting different people together to co-create a product is the best way to discover your unknown unknowns. If the risk and cost of failure is high you may want to consider this.
Wrap-up
Many things contribute to the order of the Product Backlog, value that makes our customers love our products more today, things that make us go faster, things allow us to focus more, things that unlock new markets. Each of these should be paired with evidence and this is where learning comes in.
Explore on, stay empirical, and favour the Return On Fast Learning (though maybe don’t abbreviate it).
