HealthTech Product Story: Survey results that almost cost us millions
As a Product Manager working on various products, I face questions daily to which I don’t have the clearest answers. Yet, I must make decisions to avoid blocking those who are developing the product with me. Often, when we are uncertain about an answer, we seek data to help us make a decision. However, as always, the question is how reliable and accurate that data is.
The next step after deciding to gather data is to choose how to collect it. Frequently, Product Managers turn to surveys as they’re fast and quantifiable. But are we using the right tool for the questions we have? And if we choose surveys, do we really consider that respondents might just randomly answer or hesitate while providing their answers? Are they trying to please the survey creator?
Surveys are preferred by Product Managers because they provide data in numbers or percentages quickly and efficiently. This makes our job easier and faster — the decision is to go with the solution that has the highest percentage — logical, right? At least for Product Managers, but not for professional Researchers who often warn that surveys are not always the best method.
Let me illustrate my moment of enlightenment when I finally understood, through my own experience, why Researchers caution Product Managers to be wary of surveys as a solution to everything.
Step 1: Choose the right method
We were working on an early ideation for a potential health tech product with a renowned institute and international surgeons. Since the product had to be developed from scratch, we were unsure about the exact measurements needed in labs to solve a particular health issue. To gather this information, we first set up ideation rounds with doctors and explored scientific literature. With a tight 10-day deadline due to external pressures and funding options, we needed quick insights from specialized doctors who were not available for lengthy discussions.
As a true Product Manager, I proposed conducting a survey. We decided to use a tool with a large pool of respondents, targeting these doctors. Collaborating closely with the Researcher, we knew that a survey wasn’t ideal, but time was of the essence.
We proposed asynchronous video surveys. This would get us fast results, but the structure would be different. The video survey, relying heavily on the think-aloud protocol, would capture not only their verbal responses but also their non-verbal cues, such as facial expressions and body language.
Step 2: Work with the Researcher to prepare everything
It took us several days to craft a great screening process for that platform and a good script without leading questions, double-barrel questions, and other common pitfalls. We tested on 3 respondents before the full launch. The Researcher added some extra flavor questions, such as “Can you describe a recent patient case where XY metric was crucial?” and “How did you decide which metrics are most important for addressing XY issue?”
By analyzing their detailed explanations and observing their expressions and tone, I could understand not just what information they thought was important, but whether and how they applied them in real-world situations, the difficulties they faced, and how confident they were in their answers.
Step 3: Don’t celebrate too early
We received the study results within two days, and at first glance on paper, I was happy because many responses aligned among the respondents and we could quickly confirm which data was needed for the second phase.
As a typical Product Manager, I would have made a decision immediately based on the percentages and results, but I decided to listen to Researchers and review the recordings myself. As I reviewed the respondent recordings, my enthusiasm waned significantly.
Step 4: Analyze it to the core
Respondents answered all the questions, but when it came to health metrics, they actually weren’t sure about their answers at all. They hesitated, often guessed, and changed their answers before confirming each one. Instead, they suggested the metrics they were most familiar with, not the ones that would actually help solve the problem the product was intended for. And remember those extra flavor questions? They provided answers that indicated we needed totally different metrics and a whole different approach to the product we were prototyping.
Given that it was a health product, there was no room for the slightest mistake, and we analyzed everything in detail. Detailed analysis, and not rushing to conclusions, saved us millions that could have been spent on the wrong product direction.
Step 5: Realization and questioning of previous surveys
At the same time, I asked myself — how many times have we as Product Managers misinterpreted results just because the percentage in the results pointed to one thing? How many times have we taken the easiest method just to get an answer as quickly as possible?
Data from surveys are like the tip of an iceberg; what lies beneath is where the real insights are found. Dig deeper, think critically, and listen to your Researchers.
This approach ensures that as Product Managers, we make informed decisions based on reliable data, ultimately leading to better products and solutions.
Special thanks to our Researcher who saved us in this case — Kate Kemeter
