The right tool for the job

4 research methods that product designers and product managers should be using in their discovery process

This article outlines four methods and some tools that we’ve used successfully at Babbel to get quick answers to whether we should pursue an idea or not. I find this is one of the largest struggles of the teams I’ve worked with: we’ve come up with ideas that look good on paper and then we default to two rather costly methods to test if we’re right: large scale interviews and A/B tests. This article presents alternatives so you can test an early stage idea fast and cheaply before you invest in more costly and higher fidelity tests.

• One-question surveys
• Low-fidelity fake-door tests
• Pen and paper prototype interviews
• Self-moderated tests

I’ve listed the methods in increasing complexity and effort, as well as reliability of the data. Why is it important to list speed, effort and reliability? Well, you want to spend little time and money when you have little confidence in a solution. I encourage you to start using the cheaper tests first to get early signals of whether your solution will work or not, and gradually invest more time and effort in testing it as you gain confidence in its chances of delivering the desired outcome.

Cheaper and faster methods usually involve more unreliable data, but you can overcome that caveat by establishing what success looks like for your assumptions to be proven true, and once you get the results, you can decide whether you move on to testing with higher reliability tools, or whether you dismiss the assumption as untrue. If you find that your critical and risky assumptions are less true than you thought, you will have to decide whether you change the solution based on that new information, or whether you move on to testing another solution altogether.

Let’s look at when you might want to use each of these methods, how long it will take you to set them up, and how long you’ll have to wait to get results.

One-question survey

💡 Good for…

Gathering early signals of confidence of your desirability assumptions (i.e. do users want or need your solution, or the different core components of your solution?).

⚒️ You need

Any survey tool that allows you to ask questions and offer different types of answer options, like multiple choice and/or free text fields.

⏳ Time needed

Within 24 hours, you can learn something critical for your solution.

• 1 hour of designer focus time to compose the survey
• 1 hour of collaborating with UXR and Content Design to revise it
• 1 hour to determine the profile of participants and event triggers
• 10 minutes of coding and launch
• 8 hours of data collection (dependent on your traffic)
• less than 1 hour of analysis

👥 What we did

In Babbel we’d come up with a solution that required users sharing their timeline to achieve a goal and how much weekly time-investment they were willing to invest to achieve it. We were assuming our users were willing to share their goal timeline and time-investment or were interested in setting them if they didn’t have these goal specifications yet. To validate this assumption, we launched a one-question survey containing two questions.

1. “Select the timeline in which you want to achieve your learning goal.”
a. Less than 1 month | b. 1–3 months | c. 3–6 months | d. More than 6 months | e. I don’t have a timeline but I want one | f. I don’t have a timeline and I don’t want one

2. “How many times per week do you want to learn to reach this goal?”
a. 1–2 days a week | b. 3–4 days a week | c. More than 5 days a week | d. I don’t have a weekly plan but I want support making one. | e. I don’t have a weekly plan and I don’t want one.

We aimed to collect the info from 1000 users, which would require between 8 and 24 hours of collection, according to our knowledge of our user traffic. We agreed that, in order for us to see the assumption validated, at least 60% of the answers would have to be distributed between 1.a)-e) and 2. a)-d) (i.e. they had or wanted a timeline and dedication). And less than 40% of the answers would be “I don’t have one or want to” (1-f or 2-e). We would carry on testing other risky assumptions of this solution if we obtained these desired results, and we would pivot if we found that more than 40% did not want to have a timeline or weekly dedication.

The result? Within 8h we already had 1000 responses, which lucky us, validated our assumption, so we moved to the next assumption the next working day.

Within 12h we had learned about a critical component of our solution, gaining a bit more confidence and swiftly moving on to the next risky assumption. I will caveat that the time to collect data, which was eight hours for us, might vary depending on how much user traffic you have, and what data collection regulations you have. One-question surveys are generally a valid method for most digital products as long as they don’t collect traceable personal data.

Low-fidelity fake-door test

💡 Good for

Cheap quantitative test for desirability assumptions. I call this a low-fidelity fake-door test explicitly to differentiate it with the more costly (and reliable) standard fake-door test.

⚒️ You need

A tool that allows you to insert a bit of text and a CTA cheaply and fast in your product. Some CRM tools might do the job if you’re already using one in-product, or any existing pop-up logic if you can reuse code or have development support.

⏳ Time needed

• 1 hour to design the test hypotheses and copy
• 1 hour to have it reviewed by content design and UXR
• 30 minutes to implement in whichever tool you use
• 12 hours to collect data (traffic dependent)
• 1 hour to analyse the results and conclude on next steps

👥 What we did

Recently we were considering different ideas to show progress to our users after they completed a learning activity. Each concept relied on different foci for progress: activities left to achieve a goal, rate of accuracy, and weekly learning summary.

We wanted to find out quickly if these concepts were desirable before we invested in any of them. We also wanted to see which of the concepts appeared to be most desirable. For that purpose, we came up with ways to convey the concept in text and a CTA to view that concept:

Screenshot of the low-fidelity fake door test

As we were tight on time, we decided we wanted around 500 users seeing the pop up, and we established we needed to see that at least 50% of users who viewed the fake-door pop-up also clicked on the CTA (click-through rate). If they clicked, we inferred it meant that they were interested in finding this information, and therefore that they found it desirable. We drafted the text, determined in which part of the user journey it should pop up, and let it run over the weekend. On Monday, my colleague Matheus had the results interpreted within an hour and was presenting the conclusions to the team in the weekly sync after lunch. He has written an article with more details on several of the most recent tests we shared. We found the three concepts were outperforming our metric, so we opted to bring the three to the next round of testing: preference tests.

Why isn’t this an actual fake-door test? Because the traditional fake-door test usually simulates the actual experience you’re exploring much closer with UI, not just copy pop-ups (unless that’s the actual solution). This is also the reason why low-fi fake-door tests are a great, speedy way to learn fast about your desirability assumptions, with minimal design and development investment.

Pen-and-paper prototype testing

💡 Good for…

Testing the desirability of your solution early and fast, as well as getting a quick understanding of your users’ mental models and expectations.

⚒️ You need

The essentials are a very basic wireframe visualisation of your concept and clear questions you want to ask your user about their first impressions of the solution you’re presenting to them, the expectations of how it works and how it may impact their experience. I recommend doing this when you have not fleshed out the idea entirely or created high fidelity designs, so you can feed the early insights you’ll gain of how your users think this solution may solve their need/pain or desire into your design process.

You don’t need expensive tools — in the past I’ve used google slides and miro board, as well as other visual design tools like Figma.

⏳ Time needed

• 1 hour to create pen-paper prototype
• 1 hour to write the interview script with our UXR
• Several days to recruitment our participants, as the interviewees needed to find a slot
• 2 hours to conduct the interviews
• 1 hour to synthesise the results synthesis

Within 1 week you can obtain early signal results whether your solution is going in the right direction, and you can get usable insights that you can feed into your design and further flesh out the solution.

👥 What we did

Picking up the solution that required users setting a timeline and time-dedication to achieve a goal, after finding most of our users do have a timeline for their goal and weekly dedication through the one-question survey, we decided to quickly wireframe in miro the main steps of the experience involved in our solution.

We wanted to test our assumption that our users wanted to reflect on the different aspects of their goal (timeframe, time commitment, activities they’ll have to complete) and that this would result in high motivation to commit to accomplishing that goal. We recruited 4 subscribers within a couple of days, and dedicated 30 minutes per participant to guide them through each step and capture their impressions and expectations. At the end of the interview, we asked them how likely they were to continue learning, and how motivated they felt about their learning experience.

With each interview, we learned that our learners expected to get a recommendation after answering these questions, not just to “commit” to their goal. They stated that the reflection process was beneficial, but the recommendation was really what would incentivise them to carry on learning. So, although this was all self-reported, we now had more insights on mental models and expectations that we could use to design an activity recommendation screen.

Screenshot of our working board that we gradually showed participants in our interview: a low-fidelity wireframe of what the solution may look like so we can capture reactions and expectations in a fast and cheap way.

Self-moderated testing

💡 Good for…

Gathering usability data without spending the time interviewing yourself. In our case, we’ve conducted task-based usability tests, 5-second tests, 1-click and preference tests to find out discoverability, understandability and preference assumptions: do users see this information, understand it and which of the variants we have designed do they prefer, and why?

⚒️ You need

You will likely need to invest in a third-party tool. We are using UsabilityHub.

Since this requires higher fidelity prototypes than wireframes, more participants (the recommended number for these tests is around 15 participants), it is a good test for your discovery phase once you’ve de-risked the desirability assumptions and you want to focus on how to make your idea usable.

⏳ Time needed

• Variable time for prototype Design. The time you’ll need depends on the size of the solution. If you’re just testing the usability of one mobile screen, you might only need one hour of design work. If you’re comparing several solutions of the same screen in a preference test, or testing a flow consisting of several screens, this could take a whole design work day.
• 1 hour of test design to write the questions and tasks.
• 1 hour to code or implement the questions and tasks in the tool.
• 24h to a week of data collection. Again, this will differ depending on your user base and traffic.
• 1 hour of data analysis

👥 What we did

My colleague Mette was recently working on a way to make our users more likely to click on a specific card in our home tab. This card is an essential part of the learners’ journey, as it takes them to a placement test so they can start from the right course according to their level. We had noticed that the card had some usability issues, so Mette designed different ways to solve that problem.

Since we couldn’t quickly test these variants in the product due to an overlap with another running test, we decided to find out in a fast and cheap way to remain unblocked: Mette and Mauro, our UXR colleague, designed a 1-click test to see which of the variants had higher chances of making users click on it over the other possible CTAs in the screen. The test went online on a Friday, and by Monday we could already see a clear winner that could be put into the development track.

Screenshot of the results of a 1-click test. The blue dots indicate where participants of the test tapped first when seeing the screen. 14/20 participants tapped in the expected location.

Conclusion

These are the four methods we use on a weekly basis to quickly de-risk our solutions and gain rapid insights without the need to invest more than a day of design and UXR work. Not only are these methods great because of the speed in gaining valuable and usable insights into the potential of our solutions, but they are also methods designers and product managers can learn relatively quickly how to use, removing dependencies from other functions like UXR and developers, and accelerating the speed at which move their discovery process forward.

We also use higher fidelity and complexity tests later in the solution discovery process, keep your eyes peeled for the next article on how to choose your quantitative method: do you always need an A/B test?

Credits

This list was inspired by Teresa Torres’ advice in her book Continuous Discovery Habits and Bland and Osterwalder’s Testing Business Ideas, as well as the experience my colleagues, Matheus Winter Dyck, Mauro Fernández, Lisa van Aswegen, the dream team with whom we explored and refined most of these tools at Babbel, and who helped edit this article. Thanks to Mette Harker for letting me show her work.

References

Bland, D.J. & Osterwalder, A. (2019). Testing Business Ideas: A Field Guide for Rapid Experimentation. Wiley.

Torres, T. (2021). Continuous Discovery Habits: Discover Products that Create Customer Value and Business Value. Penguin Random House.