The Experiments Backlog: Unleash The Einstein Inside You
The Product Backlog is the single most important artefact of mature product teams. It is the source of truth. Teams invest a lot of effort in discovery, analysis and design to keep it just that. It is the ultimate representation of the product owners mind, which then gets manifested into a product loved by users. Hopefully!
However, startups build products organically based on customer feedback and constant business model validation. Everything a startup does is an experiment, based on a hypothesis, waiting to be validated. In the Lean Startup, Eric Ries points out that an important goal of a startup is validated learning. The learning that comes out of running experiments and measuring response.
The Experiments Backlog is to a startup as a Product Backlog is to a product team. But it doesn’t have to only be used by startups. It can be used by high performing product + engineering teams that run many experiments a day.
Simply put, it is a backlog of experiment stories. A single source from which experiments can be planned, tracked, validated and archived. A startup can have many of them ongoing at any time. Usually, this results of these experiments will lead to changes in the business model canvas (or lean canvas) without really being recorded for posterity.
But the business model canvas is a living document that morphs over time. It’s hard to look back and figure out which experiment led to a decision a year or so down the line.
That’s why experiments need to be recorded, prioritised and planned from one place. Each experiment is expressed as an experiment story in the backlog.
Personally, I prefer Alex Osterwalder’s test card format for the experiment story —
We believe that [hypothesis]
And to verify that, we will [run this test/experiment]
And measure [the unit of measurment]
We are right if [success criteria / threshold]
It lays it bare; what we have is only a belief until proven. It is important to define the success criteria beforehand. To avoid post-experiment rationalisation.
For example,
We believe that if we highlight the CTA button, then we’ll have an increase in conversion
And to verify that, we will change the colour of the CTA button to red
And measure conversion rate
We are right if it increases to 20% from 10%
Another attractive alternative is Jeff Gothelf’s experiment story format from the book Lean UX —
We believe that [building this feature], [for these people] will achieve [this outcome]
We’ll know we are successful when we see [this signal from the market]
Quite similar. If an experiment calls for coding new features, the user or job story can be fit inside the ‘And to verify that, we will build a feature…’ section. And the acceptance criteria and validations can be specified as in a normal user or job story. Or as Jeff Gothelf suggests for UX, you can have two backlogs — an experiment backlog and a delivery backlog.
The experiment backlog bakes the build-measure-learn feedback loop right into the day to day process of product development . An experiment mindset is a growth mindset, and the experiments backlog serves as a representation of that.
It drives Continuous Learning. The sooner you test your hypothesis by putting it in front of your users, the sooner you can find out how valid your product choices have been. By coupling it with Continuous Delivery, the engine that enables Continuous Learning, you can figure out how your users are engaging with your product through quantitative analysis.
Everything is an assumption until you test it. Happy experimenting!
P.S: If your friends will benefit from this article, do share it. Thanks!
Further reading on this topic here:
Alex Osterwalder : Validate your ideas with a test card
Everything a product manager needs to know about experiments
Read some of my other articles here:
