Experiments and validating hypotheses in product creation (digital bank case example)
In this article, we’ll dive into the world of hypothesis testing with practical examples, drawing insights from a case study involving a digital banking app. Learn how to effectively formulate, test, and validate hypotheses to drive innovation and enhance your product’s success.
Step by step, we’ll dissect the process:
- Why is it important to write and test product hypotheses?
- How to write effective product hypotheses
- What experiments could you use for your product
- How to use the results of product experiments to improve your product
The Power of Testing
When you’re creating a product, you’re bound to have a lot of ideas. But it’s important to remember that:
Any idea or information is only hypothesis until it is tested and verified in the real world. Even if it comes from an authoritative source, it is not a fact.
Building a product on assumptions is like walking on a tightrope without a safety net. Just as a misstep on the tightrope could lead to a devastating fall, acting upon unfounded assumptions can result in detrimental outcomes.
To create a product that truly satisfies the needs of users and the market, you can’t go on a whim, because the cost of a mistake can be too high. So what do you do then? Understanding your customers and running experiments to test product hypotheses is an essential part of product development. It allows teams to make more informed decisions about how to build and improve their products.
It helps:
- To reduce risk: By testing hypotheses, product teams can minimize the risk of launching a product or feature that is not well-received by users or that does not meet business goals.
- To make data-driven decisions: Allows to make decisions based on objective data, rather than intuition or gut feeling. This leads to better products and features that meet the needs of users and businesses alike.
- To improve efficiency: Helps avoid building features that users don’t want or need. It also helps them identify and fix problems early on, before they become costly or difficult to fix.
- To foster innovation: Encourages product teams to experiment with new ideas and approaches. This can lead to new and innovative products and features that delight users and disrupt the market.
Process overview
Product hypothesis writing and testing is a powerful tool that can help you make better decisions about your products. By taking the time to test your ideas, you can minimize risk and increase your chances of success.
The overall process can be summarized as follows:
- Collect your hypotheses: What are the key assumptions that you are making about your product, service, or market?
- Rank your assumptions in order of importance: Not all hypotheses carry the same weight. Some, if disproven, could have a more profound impact on your business compared to others.
- Design appropriate tests: To validate or invalidate your hypothesis, the nature of the test will depend on the hypothesis itself. When designing your experiments, establish clear metrics and success criteria. This will enable you to objectively assess whether the data supports your hypothesis.
- Conduct the tests: With your test design complete, it’s time to put it into action.
- Analyze the results and Capture Learnings: Once your tests yield data, carefully examine it to determine whether it validates or refutes your hypothesis. Then, plan the next steps accordingly.
Hypotheses collection
Good ideas come from good observations. To keep your product development and innovation wheels spinning, you need to find problems that are worth solving.
Here are some sources where you can get good ideas for your product or service:
- Market research results: Help you identify trends and changing consumer needs, facilitating proactive adaptation and innovation.
- Competitor analysis: Helps you understand what your competitors are doing right and wrong, and find your own way to develop your product.
- Customer interviews: Help to gain insights into user behavior and preferences, thereby understanding their needs and challenges.
- Insights from support team: Support staff are constantly interacting with customers. They can provide you with valuable insights into the problems that customers are facing.
- Idea in progress from someone on the team: Don’t forget that your employees may also have good ideas for hypotheses. Encourage them to share their ideas with you.
- Brainstorming: Involve a wide range of people in brainstorming to get diverse perspectives.
Hypotheses should be based on a deep understanding of the target audience. The more the team is prepared and has researched the audience through qualitative methods (interviews, observations, diary studies, etc), the better the hypotheses and experiments will be, which will bring more value.
The case
Let's say our team is facing a challenge: how might we increase engagement on Saving account in our digital bank app? To get started, we conducted a series of interviews with customers to understand their pain points. Based on what we learned, we brainstormed some ideas for new features that could be helpful.
Next, we move all our ideas into a Notion database, a centralized location for storing and tracking product hypotheses. This allows us to easily monitor progress, manage the backlog, and organize hypotheses through helpful tags. This is a living document that is constantly being updated. It is an important resource for the team.
Prioritization
After we’ve come up with a bunch of hypotheses. How do we know which ones to test first? That’s where prioritization comes in.
Prioritizing your hypotheses means ranking them based on how important they are to your business and how risky they are to test.
RICE
One popular method is to use the RICE framework. RICE stands for reach, impact, confidence, and effort.
- Reach: How many people will be affected within a given period of time?
- Impact: How much of an impact will this hypothesis have on your business if it’s proven true? (Massive = 3x, High = 2x, Medium = 1x, Low = 0.5x, Minimal = 0.25x.)
- Confidence: How confident are you that this hypothesis is true? (High = 100%, Medium = 80%, Low = 50%.)
- Effort: How much effort will it take to test this hypothesis? (Effort is estimated as a number of “person-months”)
After evaluating all the factors, use this formula:
Once we’ve scored your hypotheses on each of these factors, you can rank them from highest to lowest priority. The hypotheses with the highest scores are the ones we should test first.
In our case, the Financial Calculator won the race! It’s pre-built, so it’ll launch fast and help our customers with their paint points.
Hypothesis Prioritization Canvas
Another popular method for prioritizing hypotheses is to use the Hypothesis Prioritization Canvas. This is a simple 2x2 matrix with risk on the x-axis and perceived value on the y-axis.
- Risk: How risky is it to test this hypothesis?
- Perceived value: How much value will this hypothesis add to your business if it’s proven true?
Place your hypotheses on the canvas based on their risk and perceived value.
Designing test
Alright, we picked an idea to test. Now, it’s time to refine it and choose a test that will help to validate it.
To work with hypotheses efficiently, use the SMART framework.
After refining your idea should be documented in a format that includes the following:
- Idea: A clear and concise statement of the hypothesis being tested.
- Test: The specific method or approach being used to test the hypothesis.
- Metric: What data you are going to measure.
- Success Criteria: Benchmark for assessing the success or failure of the hypothesis.
- Execution dates: Start and End date of the test and the overall time required.
- Owner: The person responsible for carrying out the test.
- Date Created: The date the idea was first documented.
This format provides a clear and concise way to document ideas and tests, making it easier to communicate and track progress. It also helps to ensure that all relevant information is included so that the test can be carried out effectively.
Exploring Testing Methods
There are a bunch of different ways to test a hypothesis. Which one you choose depends on the hypothesis, your budget, how much time you have, and how accurate you need the results to be. If you are curious about choosing the right testing method, check my other article Selecting method for testing product hypotheses: a practical guide
Here are a few most common experiment techniques for digital products:
- A/B testing: This involves creating two different versions of your product and then randomly showing them to users to see which one they prefer. This can be a great way to test out different design ideas or features.
- Analytics: This involves collecting data on how users are using your product. This data can help you identify trends and patterns in user behavior and make informed decisions about your product development.
- Usability testing: This involves watching users as they try to use your product or prototype to see if they can find what they need and complete tasks easily.
- Surveys: Distribute questionnaires to a broader audience to collect quantitative data on user demographics, preferences, and attitudes toward your product ideas.
- User Interviews: Engage in one-on-one conversations with potential users to gather insights into their needs, preferences, and pain points. This qualitative approach helps understand their motivations, behaviors, and reactions to your product concepts.
- Landing Page Testing: Create multiple versions of your landing page and compare their performance in terms of conversion rates, click-through rates, and time spent on the page. Identify the most effective landing page elements and messaging.
- Paid Advertising Testing: Test different ad copy, visuals, and targeting parameters to optimize your paid advertising campaigns. Identify the most effective ad combinations that drive clicks, conversions, and ROI.
- Mock Sales: Measure customer interest before investing in developing your full value proposition. The idea is to create a “ghost product” and see how many users will try to engage with it. This approach can be implemented both online and offline, allowing you to test different levels of commitment and gather valuable data about pricing and demand.
- Canary Testing: This specific technique is where a new feature is released to a very small percentage of users. Typically monitored closely for bugs or performance issues before wider rollout.
The most important thing is to test the hypothesis as quickly as possible without spending too much time or money
To maximize efficiency and resource utilization, start with the simplest and most cost-effective tests first. If you achieve a positive result, you can then consider further testing to refine your findings and gain greater confidence, especially for critical hypotheses.
Consider leveraging pre-made solutions readily available. Developer resources are valuable, and utilizing existing services, engines, scripts, and other pre-built tools can significantly expedite your testing process.
To test our idea that financial calculators will increase engagement on saving accounts, we have decided to integrate a few basic calculators into the app. Additionally, we will add a call to action for setting up automatic top-up transfers. We will track the calculator open rate, CTA click-through rate as well as the number of users who have set up automatic transfers to see if our hypothesis is true.
When deciding on the metric for your test, remember to set a relevant timeframe. Keep in mind that some metrics may take longer to show significant results. Therefore, it’s important to be patient and allow sufficient time for users to interact with your product before drawing conclusions.
Learning
The test is over, the data crunched, and now comes the moment where everything clicks into place: the learning stage. Here, we step back, break down our observations, come up with conclusions, and chart the course for the next steps.
The findings from our testing efforts can steer us in three different directions: Hypothesis Invalidated / Need More Investigation (Learn more) / Hypothesis Validated. Depending on these conclusions, we may need to consider various approaches for the next steps:
1. Hypothesis Invalidated:
The data speaks, and your hypothesis stands disproven. While this may seem disappointing, it is a valuable learning experience. Here’s how to navigate this path:
- Dig into the data: Look closely at the results to see if there are any patterns or surprises. Maybe it didn’t work for a specific group of users, or something unexpected happened.
- Rethink the idea: Was our original guess off? Maybe the test wasn’t set up right? Going back to the drawing board and refining our understanding of the problem can help us get back on track.
- Pivot and iterate: Don’t throw the baby out with the bathwater. Perhaps a slight tweak to the idea or test approach could lead to success. Explore potential modifications and consider further testing cycles. Be willing to pivot and adapt your approach based on the new data and insights gained.
Imagine we tested our hypothesis, and we found that the number of customers who set automatic transfers remained unchanged. In this case, we capture our learning in the hypothesis card, change the status to “Invalidated,” and fill in the Notion card with the necessary information.
2. Need More Investigation:
The results are inconclusive, leaving you on the fence. This is often the case with critical hypotheses or complex test scenarios. Here’s how to proceed:
- Increase sample size: Gather more data by expanding the test audience or extending the test duration. This can provide a clearer picture of the true impact.
- Run additional tests: Consider using different test methodologies or targeting specific user segments to gain further insights and eliminate potential biases.
- Conduct qualitative research: Supplement quantitative data with user interviews, surveys, or usability tests to understand the “why” behind the results.
Suppose the test showed a slight increase in customer activity, but not as much as expected. This suggests the hypothesis is partially true, but requires further investigation.
3. Hypothesis Validated:
The evidence is clear, and your hypothesis has emerged victorious! This moment calls for celebration, but also for careful consideration. Here’s how to move forward:
- Develop a rollout plan: Determine the best way to implement your idea into the product, ensuring a smooth and positive user experience.
- Monitor and iterate: Track the impact of the change and be prepared to adapt based on user feedback and performance data. Remember, no solution is perfect, and continuous improvement is key.
- Share your findings: Disseminate the knowledge gained from the testing process within the team. This can help inform future decisions and encourage a culture of experimentation.
After conducting thorough testing and evaluation, the hypothesis regarding the tested idea was successfully validated. The results indicated that the idea had a positive impact and showed promise for improving user engagement.
Summary
Validating hypotheses is an essential part of the development of any new product or service. By following the steps above, you can minimize risk and ensure that you are working on something that can really succeed in the market.
Main takeaways:
- It’s dangerous to rely on intuition
- It’s important to come up with hypotheses in a systematic way and to test them rigorously.
- Focus on ideas that are easy to test and have the potential to make a big impact.
- An experiment is not the end of the process. It’s just the beginning of a continuous cycle of improvement.
Save time and stay organized with my customizable Notion template for tracking project hypotheses. It’s free to duplicate and use! Link to template
