The Mom Test, chapter 2: avoiding bad data

When you’re talking to a potential user about your startup (or startup idea), there are all sorts of ways you can get bad data by asking the wrong sorts of questions. The author of The Mom Test, Rob Fitzpatrick, shares some insights.

Before I dive into those, I wanted to share a tip from a friend, Sumit Datta, founder of dwata. I had mentioned that I wanted to do user research, but I didn’t want to come across as spammy by asking people in forums to give me feedback on my startup. He recommended that I should just focus on the problem I’m trying to solve — people are much more open to answer questions about problems they’re dealing with.

Fitzpatrick warns about three categories of bad data you’re likely to get when interviewing potential users. The first is compliments. They’ll compliment you on your product because most people will want to make you happy, or will want not to offend you, or will want to tell you what they think you want to hear so you’ll get out of their hair. Because of all these motivations, you can’t trust them when they tell you think your product is “great”.

Furthermore, even if they’re sharing their honest opinion, you still probably can’t trust it. As he writes: “With the exception of industry experts who have built very similar businesses, opinions are worthless. You want facts and commitments, not compliments.”

Fitzpatrick recommends viewing compliments as a warning sign. Compliments mean that you may have been accidentally seeking out approval, or you may have slipped into pitch mode. The moment you recognize a compliment, you can get the conversation back on track by focusing directly on concrete details of the problem:

Them: “Whoa, cool idea! I can see how that could be really successful.”

You: “OK, but given that this doesn’t exist yet, what do you do right now to deal with this issue?”

The second category of bad data is fluff, which itself comes in three categories:

• Generic claims, like “I never pay for apps”
• Future promises, like “I would totally pay for something like this”
• Hypotheticals, like “I might use a service like yours”

To me, the last two seem like the same category. I’d repackage the categories as (1) generalizations about past behavior, and (2) speculation about the future or some hypothetical reality.

I’d guess that generalizations about past behavior are at least somewhat reliable. But it’s worth digging deeper, because there could be exceptions:

Them: “I don’t play online games.”

You: “So you don’t generally play games…when was the last time you did, though?”

Them: “Well, I guess I was playing such-and-such game online with my coworkers at lunch for a few months.”

Speculation is much less likely to be accurate, because people won’t feel bound to their answers, so they won’t bother to try to be accurate.

Fitzpatrick points out, though, that questions that lead to fluffy answers aren’t necessarily bad. As long as you remember not to rely on the answers, the questions can be useful ways to lead into more concrete questions:

You: “Do you ever have trouble keeping in touch with your friends?”

Them: “Yeah, I’m always feeling like I’m not reaching out often enough.”

You: “Can you tell me about the last person you felt like you really fell out of touch with?”

One way to deal with someone’s claim about the future is to put it to the test by asking for some sort of commitment. If they say that they’d definitely pay for your idea, you could ask if they’d like to pay $10 for the premium version of the service you’re launching next month. (Don’t worry if you aren’t ready yet; if they say yes, you can just tell them you’ll ask for their credit card info as soon as the service is ready.)

This seems like a risky move to me. People don’t like getting caught out in exaggeration, so you risk making them embarrassed, which could burn some goodwill. But I could imagine using it in scenarios like these:

• You think you can do it without challenging the person directly.
• They’re insisting on laying on the compliments, and you’re having trouble getting them back onto concrete topics.
• You’re already nearly done, so you’re willing to risk them getting upset.

The third category of bad data is ideas. People will often want to suggest features you should add, adjacent markets you should try to expand into, or ways to pivot your idea into something different. (As someone who participates in discussions with other founders, this is a serious problem — we all have lots of ideas and want to share them with each other constantly.)

Some of the ideas might even be good! But most successful startups are based on one core, important concept that has all the startup’s energy behind it. You don’t want to focus on building a million features; you want to find the most important thing and focus on that.

One thing I’ve noticed: People will often suggest lots of ideas if the idea doesn’t resonate with them, but they don’t want to offend you. They feel like they’ll waste your time if they just tell you, “Yeah, this isn’t really a problem I struggle with — I don’t think I need a product like yours.” So instead, they shift into brainstorming mode, and try to help you figure out how to make your product succeed by suggesting features it “should” have.

I once watched a Youtube video (I can’t find it now) with this gem: Don’t ask why they want X feature; instead, ask when they wanted it. Here’s the example they gave:

They were providing some project-management software. A client asked if they could implement access controls, such that there could be admin users, editors, viewers, etc. She said she wouldn’t renew unless they committed to implementing it. The founders thought about it, and realized it would be a huge amount of work — they’d have to update several new entities and relationships in the database, they’d need several new pages, they’d need to add validation logic, and they’d need to rethink lots of parts of their app, which up until then assumed that all users would have the same privileges.

So they asked her when she had wanted that feature. She said she had a contractor who had archived a project. He thought it would only archive in his view, but it had archived it for everyone. She had logged in to find the project missing, and it took several hours of frantic investigation to finally figure out what had happened. She wanted to be able to set her contractors as a non-admin user, who wouldn’t have the ability to archive projects.

The founders proposed to her that, instead of implementing access controls, they just implemented better UI that explained the consequences of taking certain actions. So a future user who tried to archive a project would be warned that it would be archived for all users. That addressed her concerns, and she agreed to that feature improvement.

In this case:

• She tried to come up with her own feature suggestion. But her feature suggestion (access controls) masked her real need (to be assured that her projects wouldn’t change unexpectedly).
• The founders got at her need by prompting her to be concrete (which is also a point Fitzpatrick emphasizes).
• The founders were able to find a better way to address the client’s need.

As Fitzpatrick writes: “Ideas and feature requests should be understood, but not obeyed.”

A final piece of advice from Fitzpatrick: try to talk less, and listen more. I’ll quote him one more time:

“After you introduce your idea (either intentionally or accidentally), they’re going to begin a sentence with something like ‘So it’s similar to…’ or ‘I like it but…’ You will be hugely tempted to interrupt and ‘fix’ their understanding. Alternately, they’ll raise a topic you have a really good answer to. For example, they’ll mention how important security is, and you’ll want to cut in and tell them you’ve thought about all that already. Both interruptions are mistakes. In each case, the customer was about to give you a privileged glimpse into their mental model of the world. Losing that learning is a shame. You’ll have the chance to fill them in later. Plus, it’s annoying to people if they start trying to help you and you cut them off to correct them.”

This, in particular, is very difficult for me. Fitzpatrick’s descriptions of why you might be tempted to jump in fit me to a T. If you have suggestions for how to avoid these temptations, I’d love to hear them in the comments!