Forming Experimental Product Hypotheses

An introduction to forming hypothesis statements for product experimentation

Chris Compston

Published in

Agile Insider

9 min readJan 16, 2019

Hypothesis Statements

A hypothesis is a statement made with limited knowledge about a given situation that requires validation to be confirmed as true or false to such a degree where the team can continue their investigation and find the best solution to a given problem.

In the most simple form a hypothesis comes as an ‘if… then’ statement, here’s a common example:

‘If I add water to my garden, then my flowers will grow faster.’

Now if this statement proved to be false and the desired outcome is that the flowers will grow faster and live longer we can add more statements to the list to either prove true or false:

‘If I move the flowers to an area of the garden with the most daily sunlight, then those flowers will grow faster’
‘If I plant all the same type of flowers together, then those flowers will grow faster’
‘If I add nutrients to the left side of the garden, then those flowers will grow faster than the ones on the right.’

Problem Statements

In the above example the problem is clearly that the flowers aren’t growing fast enough. The hypothesis statements were then created and experiments run to attempt to solve this problem. Moving the flowers, growing flowers together and adding nutrients are all experiments to prove which of the solutions will ensure that the flowers grow faster.

It’s important to identify an actual known user or business problem to form hypothesis statements from. Although hypothesis statements are an exercise in educated guessing, without a focus point it’ll be hard to identify when the hypothesis statement has been proven true or false.

The problem statement should follow the ‘How might we…’ (HMW) format and be based on actual user research findings, data driven themes or insights. The HMW format allows the team to both understand that there is a problem to solve and also helps to start the discussion that will frame their potential solution.

Examples

‘How might we help the garden flowers to grow faster?’
‘How might we increase the lifespan of the garden flowers?’

Hypothesis Statement Templates

Product Hypothesis statements can come in many different forms so pick what’s most comfortable for the team and business to understand. However they should always include the following key details:

feature/solution
user group or persona
user benefit
business outcome

A simple hypothesis statement might look something like this:

We believe that [business outcome]
will be achieved if [user]
attains [benefit]
with [feature]

Something a little more detailed with more of an emphasis on the solution:

We believe that [building this feature]
[for these people] will achieve [this benefit].
We will know we are successful when [outcome from the market].

Forming the Hypothesis

With a user or business problem to solve and a hypothesis template ready it’s time to fill in the statement blanks. As shown in the table below there’s four initial sets of detail to understand (Problem, Users, Benefits, Features) to form the hypothesis statement. The final two parts (Assumptions and MVP) will identify how best to experiment to validate the hypotheses created.

Problems
As mentioned previously the business or user problem should be formed as a HMW statement and should be initiated from previous discovery or research work run by the Product and Design teams.

Measures of success are going to be dependent on current business measures, data analysis or direct qualitative feedback from a number of users. These are often that hardest part to define for the hypothesis statements and this is usually due to a lack of business input or, little or late data analysis.

Without good measure of success it’s impossible to know whether the experiments has proven the hypothesis true or false so it’s worth spending time upfront to fully understand them.

Users
If the team doesn’t already have personas or an understanding of the main user groups affected by the problem statement then run a proto-persona session. If personas already exist but some more detail around their specific problems in a given situation is lacking then an empathy mapping session would work well. If there’s a genuine lack of knowledge about the way users are interacting with the product or the overall service then run a customer journey or user story mapping session.

Benefits
Using the understanding gained from the user identification sessions and the problem statements identified try to understand what benefits users are actually looking for. What are the customer needs for specific points in their journey? Where there are problems highlighted, what is it that users are lacking or trying to achieve but can’t?

Features
There’s many ways to ideate on potential solutions, this part of the process should only really be conducted once the former stages have been highlighted. However it’s equally important that past ideas aren’t left to expire, any ideas the team have had in the past should also be presented here. A quick way to get going with ideation is this One Hour Collaborative Workshop.

Filling in the Blanks

Spend time to consider as many possible options for each section in the table. Write down on sticky notes every possible permutation of the problems the team has uncovered, what type of users could be affected by any of those problems, what benefits will improve their experiences and all the solutions from the ideation sessions the team has had.

There are many tools, templates, sessions and canvases that can aid with completing these sections of the table. Here is a non-exhaustive list of things that might help: Gamestorming, Impact mapping, Lean Product Canvas, User story mapping, Customer journey mapping, Event storming, WhoDo, Protopersonas, Empathy mapping.

Completing each section does not have to go in this particular order, it might be that problems are drawn from investigating users and their interactions with the products.

Creating the Statement

With the table now full of ideas and findings start to line up problems and successful outcomes with the users impacted, the benefits they’ll gain and the solutions that will hopefully bring them.

It might be that some details have to be duplicated; maybe a problem affects more than one user type or a benefit can be found for multiple problems. Duplication is fine, just make sure that only one sticky per section exists per row as these will form the hypothesis statement, as is shown here:

At this stage everything is in place to create the hypothesis statement, lets see an example:

Problem statement:
‘How might we increase the number of overdraft applications on mobile?’

Template:
We believe that [business outcome]
will be achieved if [user]
attains [benefit]
with [feature]

Hypothesis:
We believe that an increase in Overdraft applications of 25%
will be achieved if customers that often go overdrawn
can apply for a flexible short term loan
with a quick apply mobile Overdraft feature.

Highlighting Assumptions

This is a pretty bold statement and it’s full of assumptions that have either been validated previously or are completely new and much less understood. The process of designing and building a new feature on mobile is both lengthy and costly to the business, how can we ensure it’s not a wasted effort?

Firstly it’s important to highlight all the assumptions the team has about this statement. Right now without further investigation we can highlight that the user affected might be those that often go overdrawn, but what if there’s many customers of this service want it simply for peace of mind? What if they want a longer term solution to their money problems rather than something short term and more flexible?

Take one hypothesis statement at a time and pick through it and highlight every assumption, whether it’s been validated already or not.

There are a few assumptions in the above statement but one of the riskiest assumptions (i.e. if we go straight into building the feature what could cause the whole thing to fail?) is that users want a way to ‘quickly apply’ for an overdraft.

This could be an assumption from prior research and partly validated, but right now it’s the one thing that if wrong would mean the time and effort it took to build this feature was wasted.

With the list of assumptions pull out the riskiest ones and order them by rank from most risky to least on the table. It’s possible to experiment and validate every single assumption the team have found, but in reality it’s generally only feasible to validate the top few riskiest to allow the team to move forward.

Experiment with MVPs

Now we have a hypothesis statement and identified the riskiest assumptions it’s time to understand how best to experiment with the potential solutions.

I believe that the concept of an MVP can come in two distinct flavours — to validate riskiest assumptions by understanding what the market wants or to deliver limited functionality for fast customer value and business benefit.

Type A: might simply be a paper prototype to test with users, a survey to gain further insight or a concierge or Wizard of Oz MVP. It might need no development work whatsoever and it’s not a basic ‘first release’ with defined future releases already on the backlog. The purpose of this is to ensure that the team can validate their riskiest assumptions before spending any further time or effort to build a real feature.

Type B: can again be a concierge or Wizard of Oz MVP, but equally it could have further analysis, design work and rapid development to get it into the market quickly. This is more useful if the assumptions are less risky (i.e. the team are more confident they’re doing the right thing based on prior knowledge or experience) and speed to market could bring greater business benefits.

It could be that there are multiple possible types of MVP experiment for each of the identified risky assumptions. However it’s always best to first test the riskiest assumption with the least effort MVP before moving on to others.

Now the table is complete we have a hypothesis statement with the riskiest assumptions identified and experiments that can validate whether the assumption is true or not.

Problem:
‘How might we increase the number of overdraft applications on mobile?’

Experiment:
We intend to validate our riskiest assumption that customers want a quick apply overdraft feature by building a simple one tap application screen that will send user details to an Overdrafts team to follow up and complete the process.

Outcomes

If the experiment shows that the risky assumption was actually validated as true then the team can move on and start to design and build the intended solution with a level of confidence that it will bring both customer value and business benefits.

If however the assumption is validated as false they can move onto the next assumption and reconsider how to solve the original problem. This is a fantastic place to be as the team has just saved the business months of development work and the associated costs.

This is the power of product experimentation using hypotheses.

If you’re looking for more about product, Agile, Lean and design you can follow me on Twitter @ndxcc or read more here.