An Honest Look at How We’re Running the Product/Market Fit Survey at dashdash
If you are a B2B company or have been following the tech Twittersphere over the past year, you’ve probably seen mentions of the mystical “Product/Market Fit Survey”. With a few short questions, it promises to be the barometer for where you are on the quest for the holy grail for early-stage startups.
“Product/market fit means being in a good market with a product that can satisfy that market.” — Marc Andreessen
Initially introduced by Sean Ellis a decade ago, the survey re-gained prominence last year when Rahul Vohra wrote a great post on First Round about using the survey at Superhuman. In short, a leading indicator of a startup product/market fit can be measured as the percentage of users who would reply being “very disappointed” when asked how they would feel if they could no longer use your product. If you’re over 40%, you’re golden. If not, you’ve got work to do.
At dashdash — a spreadsheet on steroids — we’ve been using the survey for over a year. We thought it might be useful to others if we shared what we’re doing, what is working, what’s not, and how we’re thinking about it. This is an honest take on the ins and outs of using this survey as an important tool in your business discovery toolkit. Let’s dive in:
Timing and Frequency
For the past year, we’ve been sending the survey every 3 months, usually in the first week of a new quarter. We chose this timing and frequency because:
1) Product roadmap: Large product initiatives are normally delivered in the latter half of the quarter, and we want to give our users enough time to experience large product updates before responding to the survey.
2) KPI mapping: We set our company, product, and business KPIs on a quarterly cadence, and the survey should be part of the quarterly objectives.
There is one disadvantage of this timing though. At the end of every quarter, we conduct our “Growth workshop”, where the Product and Business teams get together to lay out the challenges to tackle in the following 3 months and analyze how the previous quarter went.
Since the survey is only sent a few weeks after the workshop, we don’t use its results directly in our quarterly planning meetings. It is one of the reasons we’re separating the Growth workshop into two sessions: the Quarter Planning, held 3–4 weeks before the end of the quarter, and the Quarter Retrospective, held 2–3 weeks after the quarter is over and all the data has come in.
Audience
After deciding when to send the survey, you’ll need to select your audience. For this, we’re only interested in feedback from people who actually used the product. We’ve defined our audience as users who were active at least in 2 of the previous 6 months. We could have used a much shorter timeframe, but since we have use cases with a daily/weekly/monthly frequency, we felt it would be relevant to gather feedback from a large pool of people.
As general principles, you’ll want to select an audience that:
1) Aligns with the natural frequency of your product: If you’re running a product with daily usage (let’s say Facebook or Gmail), selecting an audience based on the activity of the last 2–3 weeks is probably enough. But if your use case is less frequent (e.g. Airbnb guests), you’ll want to widen the audience.
2) Gives room for churned users to participate: You’re also looking for responses from people who used the product but were not retained. For that, it is important to define an audience that encompasses people who have used the product but left.
Format
As for the format, we keep it pretty much true to the original version by Sean Ellis. We use Typeform to collect the answers, and link them to a Google Sheet (soon a dashdash spreadsheet 😊).
Here’s a quick recap of the questions:
- How would you feel if you could no longer use dashdash?
a) Very disappointed
b) Somewhat disappointed
c) Not disappointed
d) N/A—I no longer use dashdash
This is where we’re looking for the magical 40% of “very disappointed” users 🎯
2) What would you likely use as an alternative if dashdash no longer existed?
3) What is the main benefit you received from dashdash?
4) How can we improve dashdash for you?
5) What type of people do you think would most benefit from dashdash?
Managing Replies
One of the toughest things is getting people to reply to the survey. Your users are probably busy with multiple projects, have dozens of browser tabs opened, and a long backlog of unread emails. Catching their attention is hard. We’ve tried multiple twists on the survey email but still, see that only 10–20% of recipients respond to the survey.
Don’t skip on a mid-week reminder. In the past 2 quarters, +65% of our survey replies came in the days following the reminder. We’re also seeing that plain text emails work better than “polished” ones (no buttons, no images, no GIFs), so next quarter we’re moving off of Mailchimp for a 100% plain text email. Plus, emails sent from the CEO/Founder resonate better with our audience.
Finally, we keep the reply-to email the same as the sender's email (we’re not afraid of getting some emails), and intentionally don’t track open rates, click rates, or collect personal information on who replied to the survey.
Output
Once the results are in, we’re mostly interested in four things:
1) How are we doing against the 40% goal?
This is the barometer of product/market fit and the overarching KPI that comes out of the survey.
2) What do we need to do to convert on-the-fence users?
As pointed out by Rahul in his article, we want to know what we are missing to turn our “somewhat disappointed” users into people that go crazy about us.
We go through the answers to “How can we improve dashdash for you?” and identify the features that might make the difference. Most of the time, these will be obvious improvements, but once in a while, you will find something new or something old described in a new way.
“More templates / use-cases”
“More data sources and templates”,
“Data visualization”
“Provide me with proposals what I can use the tool for besides the use cases I already do with dashdash.”
3) How do our users explain the benefit of using dashdash?
Here, we’re looking for patterns in answer to “What is the main benefit you received from dashdash?”. There is nothing better for positioning and messaging than hearing your users describe your product and what it does for them. It is a great — and often humbling exercise — that helps us craft product messaging that resonates with our audience.
“Automation of repetitive searches — Time saved!”
“Integrations made easy”
“Building spreadsheets that would be impossible to build quickly without the easy API access.”
“Still haven’t figured that out.”
4) Who do our “not disappointed” users think would benefit the most from using dashdash?
Quite often, you’ll attract users that are not a great fit for your product: they might be the wrong persona, they got into the product too early, or they just didn’t need your product bad enough. Nonetheless, these users can offer good feedback on who they think the product is for.
This is what we’re looking for in answer to “What type of people do you think would most benefit from dashdash”: is there untapped potential in going after a new user persona?
We found that unless you have a very large number of responses (>100), doing refined user segmentation is not incredibly helpful. Each additional segmentation will reduce the number of answers you’re looking at, and you will end up with a sample size too small to generalize any findings.
The output of the analysis is compiled in a short one-pager. We share it with the whole team on Slack and present it in an All-hands meeting 🙌.
Conclusion
The PMF survey has been an invaluable tool at dashdash. It gives us a pulse on how our users feel about the product, a target to chase, leaves us with areas to double down on, and occasionally surfaces new market opportunities. Hearing first hand from our customers about the benefits they see in dashdash and areas of improvement is also a great motivator for the team. And in a time where we’re all working remotely, sharing what’s going on across the company is more important than ever.
That being said, my advice would be not to lean too much on the survey results to drive decisions, but rather to inform them. There is a big selection bias in the results — you’ll only get feedback from the small percentage of people who actually responded — and it is easy to get infatuated with the enthusiasm of your most cheerful supporters.
We’ll keep running this survey in the next few quarters and for as long as we feel it is worth it. If you have any feedback, comments, or suggestions (tips on improving the reply rate would be nice), reach out.
dashdash is the spreadsheet for builders. Automate your processes with powerful Sales and Marketing tools that integrate with your stack. Currently in beta. Find out more on our website and sign up for our waitlist.
