6 steps to take you from idea to customer interviews

Karina and I recently quit our jobs, reduced our commitments and dedicated ourselves to finding a start-up idea to work on. For the past month, we were based in Russia and conducting our search from there. During this time, we got the opportunity to experiment with various techniques for remote customer development.

This post will guide you, the lean startup practitioner, through the earliest stages of customer development interviews. I will be covering how you can organise your work into sprints, find customers, filter them and, finally, schedule them in the most time-conserving manner. However, I will not be talking about the actual interview process, as this has been well covered by others.

And all this without having to physically step out of your home, particularly useful when it’s -27℃ outside!

Depth or breadth?

So you have an idea and it is ready to be crushed by potential customers. Interviewing a customer thoroughly takes time. In 15 minutes you will only scratch the surface of the problems they are facing. To go deeper will require at least 30 minutes or an hour of conversation. So, given we have a fixed time budget, should you be interviewing many customers in a shallow way (broadly) or few customers deeply?

Our interviewing methodology must be lean, allowing us to iterate quickly. A deep strategy is better suited, as it saves time and money finding interviewees and scheduling by allowing us to reject flawed assumptions as soon as possible. For example, if your assumption is that more than a ⅓ of PhD students have difficulty picking a proposal topic, interviewing just 5 people is enough to prove you wrong. If none of the students mention that they have the problem then you can be 87% sure that your assumption was flawed (this statistic was worked out using a binomial distribution calculator). On the other hand, proving your original assumption about PhD students is a much harder task as it requires a larger sample.

Furthermore, interviewing more customers has diminishing returns in the amount you can learn. Assuming the interviewees have homogeneous problems, your discoveries will start to overlap as more people are interviewed. This is why only 5 interviews is commonly recommended in UX design (reach Jakob Nielsen’s research for an in-depth explanation). However, to achieve the desired homogeneity one must be able to pick only the niche customers to interview. We will talk more about the filtering step later in the article.

In conclusion, deep interviews are most appropriate when 1) collecting qualitative data, 2) interviewees are believed to experience a similar pain and 3) you wish to discard customer hypotheses quickly. However, when collecting quantitative data, more interviews will give better results.

Finally, interviews are not the only way to probe your ideas. If you are lucky to have access to to users (who you believe to be relevant), you can study their real time behaviour using tools such as Optimizely or CrazyEgg. Such data is not only fast and cheap, but may also be of greater value, as actions are better indicators or potential behaviour than words. If you don’t have this luxury, you may still be able to overcome the shortcomings of broad interviewing by conducting a cheap survey.

To get the best of both worlds, I prefer a hybrid approach combining a broad screening survey and a small set of deep user interviews. I will discuss this approach in depth in the following sections.

Step 1: Set an interview day

So we want a lean process that we can iterate through quickly in our hunt for ideas and problems. Sounds like an excellent place for a sprint! The process I’m about to outline is loosely based on Google Ventures’ research sprints.

It’s very tempting to interview each customer immediately after you find them. However, fitting scheduling and user finding around an adhoc interview schedule is messy and wastes time. This is why I suggest you schedule interviews in blocks, all in one day.

One way of interviewing efficiently is committing to a sprint. GV use a 4 day sprint which is a good starting point because it keeps you moving quickly. Spend most of the sprint finding customers and scheduling and interview them back-to-back on the final day. This way, you have a hard deadline, so that by the end of the interview day you know if it’s time to pivot or iterate. You also don’t waste time between interviews. Bingo!

Step 2: Create a screener

Now that you have a structure, how do you find the right customers to interview? Ideally, you only want to speak to relevant customers. Have you ever got 2 minutes into an interview only to realise that they aren’t your customer?

One way of filtering your potential candidates is a screener survey. This survey will spare you the remaining 18 minutes of polite conversation: you now only have to interview a few quality customers (a sweet compromise between depth and breadth).

A screening survey I used when researching an idea in the area of postgraduate research.

To build yourself a time-saving screener:

1. Using your customer assumptions, identify which users you want to include or exclude, e.g. young PhD applicants who find it difficult to decide on a topic. Write these down.
2. Next, quantify these:, e.g. “between the ages of 18–25” or “Rates 7 or more on a scale of 1–10 for pain”.
3. Now write survey questions taking care not bias the candidate (there is a whole book about this, The Mom Test). Dan Shapiro has a good post on how to build a survey. For my survey I used Google Forms.
4. Prefix the form with a little about yourself and why your motivation to avoid misleading candidates.
5. At the end of the survey ask them if they wouldn’t mind being contacted to schedule a phone interview. To sweeten this up you could add an incentive. I offered the chance to win a £10 Amazon voucher. Not much, so you may find you don’t need an incentive at all.
6. If they agree, you should take their email address and phone number.
7. Don’t be afraid to refine the questions and even the assumptions as answers come in. If you can avoid wasteful interviews on interview day, then adjust the form.

Step 3: Advertise

Now is the time to drive candidates to the screener in any way possible. Start by leveraging your social capital as this is often cheap and easy. However, if you are constantly testing ideas, beware of exhausting your friends with requests for interviews.

Advertising is probably the cheapest (depending how much you value your time) and fastest way to acquire leads and, fortunately, the internet is full of such opportunities.

mTurk

Before discovering the wonders of screening surveys, I tried a method proposed by Justin Wilcox over at CustomerDevLabs. His post describes how one startup was able to use mTurk (Amazons micro-working platform) to interview 100 customers in just 4 hours.

Tempting, right? I couldn’t resist the lure and posted a task on mTurk asking people to call me (as instructed by Wilcox) and patiently waited. Inspite of experimenting with time zones, I only got one call after 6 hours of trying and even then, I managed to miss it!

So what was going wrong? It seems to me, people don’t like picking up the phone to a stranger. Furthermore, it’s possible that my target audience, PhD students, weren’t well represented on mTurk. Be sure to research mTurks demographics if you want to try this method. However, the risk is low given you only for actual interviews.

I never went back to mTurk with the screener, having concentrated on Twitter instead, but it might be worth another shot in future. Surveys seems to go down rather well on mTurk.

Facebook and Twitter

Facebook and especially Twitter are the new kids on the advertising block. They work much like Google Adwords with a bidding-based payment model and targeting. As social networks, however, they have a major targeting advantage.

Facebook is especially good at this. Using their advertising tools, you are able to target your adverts very precisely to people with particular interests, education, sex, age and even the device they use. This can be super useful for inexpensive interview candidate acquisition. Twitter is also able to target quite well by keywords but Facebook really has the upper hand here.

We found that, with Twitter, more people who followed the link to the survey actually filled it out, by a significant margin (11.5% — 28.9% versus 1.2% — 9.5%). It’s possible that my higher interactivity on Twitter was really the cause. Otherwise, the cost per click was similar for Twitter and Facebook (between £0.04-£0.20).

Others

I also tried using Craigslist UK but sadly got no responses. I presume that the US site gets more usage that it’s UK sister. I also read that Google Adwords is a pretty mature marketplace (read: “expensive”) so didn’t try it. However, it does cover a great deal of the web which Facebook and Twitter do not. I will definitely be experimenting with it next time.

Step 4: Socialise

What a social network provides over traditional web advertising is the ability to socialise (duh!). Not only do you get people clicking through to your link but also retweets, replies and likes. All of these you can (and should!) follow up with.

In our case, these warm leads accounted for the majority of people who filled out our form. Communicating one-to-one (even if only with 140 characters) gave people a sense of rapport. They felt they were doing me a favour and that I really cared for their answers.

While both Facebook and Twitter have built-in virality, Twitter is richer with its retweets and stars versus Facebook’s likes. These viral mechanisms give you “organic traffic” which basically means free! As much as possible you should encourage this virality. For example, include a request to retweet your ad copy or give prizes to people who retweet the most.

If you approach a stranger on the internet and ask them for a 30 minute interview, it’s very likely that you will be ignored or rejected. This is where the screener’s hidden benefit comes in. Thanks to the Ben Franklin effect, if you ask them for a small favour first (the survey), they are much more likely to grant you an interview. This, perhaps, was missing in my mTurk approach.

Step 5: Experiment

The characteristics of each advertising platform will be different depending on the target market you are aiming for. If you are looking for American housewifes, then mTurk could be the way to go. Looking for grandparents? You probably shouldn’t advertise on Twitter.

The key here is experimentation. Start by making a guess at who your target audience is and where they are located. If you aren’t getting as many forms filled out as you like then start to investigate other options (as a rule of thumb, you should probably be paying around £0.10 per click and 10% of those clicks to fill out the form). Most advertising platforms allow you to set budgets as low as £1 per day, so it doesn’t hurt to try them out.

If you come from a Lean Startup background you might be tempted to split test your advertising copy. Before you do this, there are a few limitations you should be aware of:

• Do you have the budget to run a split test? In order to have statistically significant results from the test, you need to have a substantial number of impressions on your ad, which will cost more money. Check out the sample size calculator by Optimizely.
• Facebook and Twitter seem to provide built-in support for split testing but beware. They will both only select the best advert from a set. This means it’s not possible to do multivariate testing. You can read more about the confusion this caused me on my Quora question (a big thankyou to Adomas Baltagalvisfor his insiteful answer).

Also, be aware that Facebook and Twitter can’t magically solve the sample size problem. They will do their best with the budget they are given, but you may not get reliable results if the budget is too low.

In conclusion

• Use well known split testing results. There is no need to reinvent the wheel, it’s cheaper to copy.
• Decide what would be most important to test and spend your budget wisely.
• Test one variable at a time. The improved conversion rate that you get from the test means that you need a smaller sample size to test the next variable.

All this optimisation also costs you time. This is the time you could spend interacting, so don’t over do it.

Step 6. Filter and schedule

Once you have answers to your screener, you can finally select participants and schedule interviews.

In our first attempt at scheduling, we used an automatic tool called PowWow. It works very similarly to Doodle but requires interviewees to enter their phone number and email address. After an interviewee books a slot, PowWow also takes care of emailing them a reminder. Perfect for scheduling phone interviews.

While this seems like a neat solution, it is hard to encourage strangers to follow such a link. Perhaps you have noticed this yourself if you ever used Doodle. It seems great in principle but in practice you find yourself chasing everybody up manually and waiting for them to respond in the lucky case that they do.

This is why I highly recommend taking the candidates’ phone numbers in the screener and calling them to schedule the interview instead of asking them to do it themselves. Although it might surprise you that asking for phone numbers up front is more effective, consider the survey participant. They have already committed to completing the survey in order to help you, so it is only sensible to make the most of it while they are engaged, rather than calling for action later.

Another related problem is selecting the participants for interviewing. As has beenmentioned earlier, you ideally only want to spend time with the most relevant customers. To identify those, I used Google Sheets’ filtering and sorting capabilities and rank participants (in my case, by the most perceived pain). I proceeded to phone them in this order to schedule the interviews. I found it most useful to suggest provisional times myself, which both renders a gap-free schedule and avoids putting unnecessary cognitive strain on the interviewees.

Although you might be tempted to conduct as many interviews as you can, I recommend setting a cut-off criteria to avoid the least relevant candidates entirely. Remember what we said about diminishing returns: only overlook these criteria if you are struggling to get anyone else. In which case, you should probably ask yourself why users are so unwilling to discuss their urgent, throbbing pain with you.

Interview!

So there you have it. My 6 step process to scheduling customer interviews. Now all you have to do is interview them: ask the right questions and get the right answers — easy right?

If, like me you don’t find interviewing and concluding so straightforward, take a look at these two blog posts from CustomerDevLabs: How I Interview Customers and You’ve Interviewed Customers. Now what?.

Originally posted at my blog, Start-up With Sam