Kanban Boards for Hypothesis-Driven Development
Build the right thing
Kanban boards help us limit our work-in-progress (WIP) and make current work-in-progress visible. Without a kanban board we often neglect to address work that is stuck or churning.
A kanban board for hypothesis-driven development helps alleviate three pains:
- It’s difficult to know which experiment has higher priority
- Results are unclear because too many experiments are running at once
- It’s difficult to know if we are progressing toward our goal
Assumptions
Hypotheses without measures or predictions are just assumptions. Assumptions come from anywhere: Conversations, observations. Assumptions can also come out of artifacts such as the lean canvas, business model canvas, personas, or activity systems.
Hypotheses
Validating our assumptions through hypotheses discourages feature bloat and helps us focus on building the right thing. Hypotheses help us gain validated learning as we transition through our primary stages of development: empathy, stickiness, and growth.
Experiment Design
Once we have captured our assumptions and prioritized them by relative impact on our business plan (if proven true/false), we can begin designing an experiment. Well-designed experiments help us test our hypotheses.
Experiment Reports
Experiment reports provide the basic framing to design our experiment. On the left side of the report we write about how we will run the experiment. And on the right side of the report we explain the results we observed by running the experiment and what we should do next.
Elements of an Experiment
This version of the experiment report was adapted from Ash Maurya’s article about applying lean thinking and Zach Nies’ article about framing build, measure, learn in an uncertain world. The former did not include enough detail and the latter required too much detail.
Background
Write a brief synopsis about what we want to learn through this experiment.
Falsifiable Hypothesis
Here’s a basic format for hypothesis (adapted from the Lean UX book)
“We believe [this statement is true].”
Experiment Method
The method describes how we will test our hypothesis. There are many methods for running an experiment.
Here are just a few methods:
- Survey
- A/B test
- Problem interview
- Solution interview
Variables & Measures
This is where we describe our predictions (adapted from the Lean UX book)
We will know we’re [right/wrong] when we see the following results: [qualitative results] and/or [quantitative results] and/or [key performance indicator change].
Experiment Results
Real learning happens when we can compare our predictions to reality. In the results section we list our quantitative results and qualitative results so that we can compare them to the variables & measures section.
Validated Learning
Was our assumption validated or invalidated? Briefly summarize what we learned from the experiment.
Next Steps
What are the ancillary insights from this experiment? What do we think we should test next? Should we keep pursuing this hypothesis or pivot? If we ran out of time, resources, or had inconclusive results why did this experiment fail? Try using five whys to get to the bottom of it.
“We can judge our progress by the courage of our questions and the depth of our answers, our willingness to embrace what is true rather than what feels good.” — Carl Sagan
Judging Progress
Give the kanban board a name that reflects the goal we are driving toward. On a kanban board for hypothesis-driven development, the basic states of experiments are new, active, and stopped. Run one experiment at a time to avoid inconclusive results and failed experiments. As assumptions are validated or invalidated, remove them from the kanban board and archive the learning so we can look back and see how much we’ve learned.
