Three Lessons for DIY Market Research with Surveys and… Reddit?
edit: This was originally posted in August of 2016. I’m writing this update in September of 2021. While I think this essay is still somewhat useful, the product I set out to build ultimately could not attract or retain users at a sustainable rate and I had to let the whole thing go about a year later. Perhaps someday I’ll get around to writing a complete postmortem, but for now, as you read this article, just be aware that this advice did not result in building a successful product.
I tried my best to heed the common advice: Talk to customers before building any product. Like many founders, though, I didn’t do enough of that before I started building. A few weeks ago, I decided to rectify that problem.
I’m building an app that is (and I’m still working on this pitch) a “gamified” social network for bands and fans. For my initial “validation,” I talked to friends in bands and friends who go to lots of concerts. The results were positive, but I still felt like I didn’t have enough to go on. I wanted more feedback, a larger sample size… in short, stronger validation.
My first thought was to put together a survey. I would ask people how often they go to concerts, then a bunch of questions about the app and my planned features, and hope that there was a strong correlation between frequency of concert attendance and what I was planning to build.
This is my story, but I hope that this advice can be of use to other people building products who want to get more feedback. In the end, I had nearly 400 survey responses, far more data than I could have collected by emailing or talking to people.
So, after some research, I created a SurveyMonkey account. “Well,” I said to myself, “This can’t be too hard, right?”
Lesson 1: Ask the right questions
Fortunately, SurveyMonkey has some templates. Unfortunately, none of them were directly applicable to what I was doing. However, they’re great guides to show what types of questions work for these kinds of surveys.
I broke my questions down into a few categories:
- Qualifying questions. These are meant to sort out different types of potential users. In this case, it was “How many live concerts do you attend every year?” It’s a deliberately specific and closed-ended question, meant to break people into groups that signal the difference between casual and hardcore fans. The options were 0, 1–4, 5–12, 13–20, and 20+.
- Closed-ended product questions. These are meant to measure interest in the product along multiple dimensions. How likely would you be to use this app? How often would you use this app? How excited are you about these various features (with a list of planned features and a rating scale for each one)?
- High-stakes closed-ended product questions. I only had two of these, but they were designed to answer the most burning questions I had about my plan. “How do you feel about the idea of earning points that you can redeem for VIP perks?” and “Would you prefer one app for each of your favorite bands, or one app that covers all of them at once?” Both of these questions could result in major changes in product direction. Actually, one of them did! But I’m getting ahead of myself…
- One open-ended question at the end. I don’t want people to feel like they have to type too much, so this is designed to collect stray thoughts and find out who is REALLY enthusiastic about the app (and why). For the most part, people who aren’t interested will write the same response. People who are interested will write enthusiastic and varied responses.
Avoid this mistake: I added a question about age group at the end of the survey, not knowing that SurveyMonkey automatically includes demographic questions at the end of the survey. So… I wasted a question.
Lesson 2: Paying for responses is risky
Once I finished building the survey, I took to Facebook, LinkedIn, and Twitter. Unfortunately, I don’t have a massive network of followers who will fill out surveys upon my command, so not much happened.
SurveyMonkey offers an interesting service that allows you to pay for survey responses at a rate of a dollar per response for un-targeted responses, or more on a sliding scale depending on your targeting and audience size.
Being impatient, I decided to spend fifty dollars to get some responses just to see what would happen. Within a few minutes I ordered up fifty responses with no targeting. I was instructed to wait up to 48 hours for responses. It was late in the evening, so I went to bed thinking I would wake up to find a few responses and that I would just continue to work on the app while more trickled in.
I was wrong.
The following morning, I was surprised to learn that my entire order had been filled. Excited, I clicked through to the results and… was extremely disappointed.
Here’s why:
To be clear, there were some positive responses. But overall, it was a total bust. I had paid fifty dollars to learn that most people don’t go to concerts, which is something I already knew.
Not wanting to completely give up on paid responses, and knowing that I could target by age, I tried again. This time I paid $105 for 60 responses limited to ages 18–34. That’s right, I went for the most desired, most expensive demographic.
Once again, results came in quicker than expected. And once again, I was (mostly) disappointed.
Clearly, targeting the “general” audience wasn’t going to work. I know there are people out there who are huge fans of bands, who are interested in what I’m offering… after all, there are a whole five people who say they would use this app “All the time” out of this small batch of responses. But that’s not enough to derive any useful information.
Lesson 3: Find your target audience, even if it gets awkward
Something had been nagging me in the back of my mind since I built the survey. You could just post this on bands’ Facebook pages, I thought. Off the top of my head I thought of a few bands that might have the right fan base to support an app. Panic at the Disco (a personal favorite), Coldplay (because they’re huge), Muse (also huge), and Mindless Self Indulgence (a band that lends itself to tons of Hop Topic-worthy fan art is, I assume, a great target for an app… and I do like MSI anyway).
I duplicated the survey, but added an additional question to the beginning: “How big of a fan are you of this band?” This deliberately vague question would allow me to post the survey framed as being for any band.
Avoid this mistake: As a side note, by this point I had already upgraded to SurveyMonkey’s cheapest paid plan to allow me to view more than 100 survey responses. It’s kind of crazy that you can pay for more than 100 survey responses, but you can’t view them unless you upgrade your plan. Since I had upgraded, I had the ability to create more “Collectors,” which would have allowed me to have separate links per post. I would have been able to see which responses came in for which bands, but I didn’t do that, so I just have to look at the answer to the open ended question to figure out which band they came for.
Anyway, unfortunately for the surveys (but perhaps fortunately for the future of my app), interacting with other fans on a band’s page is less than optimal. The only way to reach an audience is to post a comment on an existing thread, and the bands I checked out were using Facebook posts far less than I expected. This is perhaps due to Facebook’s algorithm tweaks designed to avoid spammy pages, and perhaps just due to a lack of interest or effectiveness. Either way, I posted links to my surveys and received very few responses.
It also felt awkward because I was posting from my personal Facebook account with my full name and photo. And there’s not really an alternative to that on Facebook, since you can’t create a new account and start posting comments all over the place without triggering a potential account ban.
So I took to a slightly less awkward place to annoy people: Reddit.
Reddit has all the right ingredients for market research: Subreddits are broken down by interest, and you can post a link along with some text, hopefully get some upvotes, and answer questions in comments.
First, I found a listing of the most popular music subreddits:
There’s a section for specific artist/band subreddits. I clicked through to the ones that I thought corresponded with the most popular bands (not artists, DJs, or rappers), and checked their subscriber counts. I was surprised that even the most popular, like Radiohead and Daft Punk, had less than 50,000 subscribers.
The next step was to post a link to my survey. But I didn’t want to just link to the survey, I wanted to provide context that might get people excited about it. So I warmed it up with the subject, “Building an app, who’s interested in a (band name) app?” It was surprisingly effective.
This is the part of the story where things start to go well.
With six posts on six different subreddits, I received a total of 201 upvotes and 250 survey responses in just a few days.
I finally found the right audience, and they were responding.
Here’s the above chart, but filtered for people who attend more than four concerts a year:
And for people who attend 13 or more concerts a year. It’s only 18 people, but as a representative sample of hardcore fans, this is what I was hoping for!
Finding the specific target audience was the key.
I realize that people who attend 13+ concerts a year are a pretty specific market, but I want to build something for super-fans first, and if it works, then consider expanding to more casual fans. (Read the awesome book Zero to One for more on this, though I will acknowledge that “another social network” is not exactly flying-car-level innovation territory!)
Bonus Lesson: Ask burning questions, get burning answers
Remember those high-stakes questions I mentioned earlier?
I had some major concerns over the idea of people redeeming points for VIP perks. This is, in my opinion, a unique and interesting idea that could define the direction of the product in the future. But it’s also potentially dangerous: What if fans get upset because they don’t have the time to earn enough points to get the perks they want? Will fans resent that the app creates a competition?
This was actually the most popular feature in the matrix of features. I added a question specifically about this idea in addition to the feature matrix question, and it was probably unnecessary. Turns out, people love the idea of earning points and redeeming them for stuff.
On the other hand, I was very, very wrong about how to structure the product. My original idea was to build one app for every band. This would have the (obvious) disadvantage of forcing people to download multiple apps to interact with their favorite bands, but would have the advantage of allowing the band 100% control over the skinning of the app. “It will be an extension of the band experience,” was my pitch.
Almost everybody hated that idea. So, six weeks into the development of my app, before launch, I started to redesign the app internally to support multiple bands with one app. My launch plan is still to pilot with just one, but after that I’ll be adding bands to the app rather than releasing more apps.
The Verdict
Running a survey to validate my idea was definitely worth the time. I probably spent a total of eight working hours on this, and a lot of it was posting on reddit, responding to comments, and looking at the results.
The roughly $150 I spent on paid results was definitely not worth the money, but I can imagine that it would be a great way of collecting results for a mass-market consumer product.
There are some caveats to this. I’m no scientist; I definitely went out looking for a result and changed my methodology until I found it. And even though the survey responses claim interest, that’s not a guarantee that they’ll actually use or like the app, even if I execute perfectly.
So, like any startup, I’ll have to forge ahead, hoping for the best but accepting that it will be an uphill battle no matter how much I think (or know) that my target market wants it. But hey, that’s what I signed up for.
