Hypothesis-driven practices to build better features

Hypothesis-driven practices can help you make better decisions that will reduce your risks while building better features

Should you really build that feature? Well, we all have great ideas for new features or improvements. Then, we implement them because we assume they will have an impact and help us reach some goals 🎯. But, we must admit that we face a lot of uncertainty in what we do.

Could you explain the path from your ideas to the solutions you deliver? Are you be able for formalize the uncertainty you face? How do you know your solution is satisfactory?

I have applied hypothesis-driven practices into my daily work as a product manager. It helped me answer questions and justify my actions. It pushed me to define measurable outcomes upfront. It taught me to formalize the unknown. And it allowed me to track lessons learned and act upon them.

In this post, you will find high-level theory about hypothesis-driven practices as well as different examples to make it all real. Finally, I’ll share 3 reasons I love working with hypotheses 🤓.

Features usually come from assumptions

An assumption is a statement for which we only have limited (to no) evidence. We tend to make assumptions based on experience, observations, data or gut feeling.

Example of an observation: many customers rate their delivery on their first order, but they often don’t answer any of the questions on the satisfaction survey. Which is a pity because this feedback is key to improving our services.

Example of an assumption: We can increase feedback (comments) from our customers on their first delivery.

Even though we cannot really work with such a high level statement, it is a great starting point for an improvement or a new feature. Isn’t it?

“Assumptions are vague, optimistic, and untestable. The vaguer they are, the harder they are to disprove.” By Tristan Kromer in Lean startup hypothesis

Let’s turn our assumption into something “usable”. To do so, we will convert it into a hypothesis. The difference between these is straightforward. A hypothesis is a statement that is actionable, and that we will attempt to prove is true by testing it with experiments 🔬.

“The starting point for designing strong experiments is being explicit about what we are trying to learn. A clear and well stated hypothesis that is testable and precise is at the heart of everything.” By Tendayi Viki & Franziska Beeler

let’s get to work.

4 step process to working with hypotheses

1. Define

When and where would you define a hypothesis?

It is possible to define a hypothesis at different granularity levels. In this post, I will focus on using hypotheses to create features 🎯.

Referring to the traditional Agile hierarchy, I like to capture new features or improvements in Epics. Hence, I believe an Epic is an ideal placeholder for a hypothesis. In fact, Epics are great to scope work and to structure it. They give vision and help teams work towards a common goal.

How would you define a hypothesis?

Obviously, there are multiple formats to define a hypothesis. The one I like to use comes from the Test Card by Strategyzer 🙌.

Start date: Set the start of your experiment. Duration: Define its length. We believe that: Describe one thing that must be true to validate your idea. Can you specify your target? To verify that: contains the test itself. What is the change you are making? And measure: What are you going to measure? This will validate or invalidate your hypothesis. We are right if: Defines your threshold. Here you’ll make a prediction. What do you expect to happen?

Also important, on the right side, you can see relative values that will help you assess and prioritize your hypothesis.

Example of hypothesis:

Here is the checklist I use to validate that my hypothesis is well written.

• Define a clear goal ✅ (increase feedback comments)
• Define a target ✅ (new customers on their first order)
• Explain how you will test ✅ (sending the feedback notification 2 hours after delivery)
• Define the expected outcome ✅ (>20% of feedback)
• Define how you will measure the results ✅ (reports with KPIs)

2. Implement

Now, we will need to implement an artifact that will help us learn whether our hypothesis is true or false. The quality of the artifact will have a significant impact on the results you will get. Therefore, designing a well-crafted artifact is key to run an efficient experiment.

Your artifact could be a paper prototype, an online survey, a hi-fi prototype, a fake lading page, A/B testing, a real feature, etc.

Early in a process, you might want to implement a cheap artifact. As you learn more, you could go for a more expensive option that will produce stronger evidence.

What makes a good artifact?

I would say that a good artifact is one that helps you get enough evidence to make a decision at the right time.

Example of an artifact: New notification with satisfaction survey sent 2 hours after delivery. I would implement it directly into production. Why? Because it would be “cheap” and results from real customers would be highly relevant.

3. Test

Testing a hypothesis actually means running an experiment. The goal of a test is to validate whether your hypothesis is true, or not. To do so, you will use your artifact and start gathering empirical evidence. Obviously, the way you run your test will depend on the artifact you decide to implement.

Example of tests: For a hi-fi prototype, you would plan interviews with users and make them test your solution. If your artifact is directly in production, you would observe key indicators.

In your test card, you defined the duration of your test. During this time, you will most probably see tendencies. Your test might take the right direction towards validating your hypothesis. Or, it might quickly show you that your hypothesis will most probably be invalid. As you are running your experiment, I would recommend you to observe the evolution of your results. You might learn things along the way.

4. Act

The last step consists of assessing the results of your test and defining your next actions. With your test, you might learn that your hypothesis was right! Or, that it was almost right. You might also learn that your hypothesis was wrong.

Example of test results: Before the experiment, only 9% (36 out of 400) of new customers would give feedback on their delivery. Let’s imagine that we ran our experiment during 3 months and that 22% (88 out of 400) of our new customers gave feedback when receiving the notification separately 2 hours after their delivery. This would validate our hypothesis 😊!

We are not not finished yet.

Every test that you make generates new insight that you can use for future product developments. This should help you define your next actions:

• What happens if your hypothesis is true? Will you scale it?
• What happens if it is wrong? Will you rollback? Will you pivot?
• Do you need more time to validate your hypothesis?

Example of next actions: Scale our solution to all the customers. Or, run the experiment a little longer to get more data.

We have now come full-circle with our hypothesis.

3 things I learned with hypothesis-driven product management

You can consider this section as my conclusion 👋.

You can formalize the unknown

Working with hypotheses taught me to accept and formalize the unknown upfront. We never start with guarantees but only with limited evidence. Our chances to predict exact outcomes are slim. It is not because a feature exists in a similar product that it will work for your product. It is not because your initial analysis and data expect your solution to reach your objective that it will happen.

It is humbling

Since I started using hypotheses, I have become less affirmative with my statements. In fact, the hypothesis format drives you away from affirmations. Instead, it helps you formulate beliefs with measurable criteria. It is humbling to accept the unknown ☝️.

Closing the loop

Hypothesis-driven product management pushes you to answer: “What’s next?”. And this is crucial. You basically keep on trying to deliver features that help you reach your objectives. You keep on running experiments until you are satisfied 😎.

Thank you so much for reading!