Balancing Customer Feedback and Data, and How Some Feedback Can Lead You Astray
Ignoring customer feedback is one of the worst things that you can possibly do for your product.
However, there is the other side of the coin, where looking too much into customer feedback and not looking into the data can result in a product that loses its core audience.
If you think that users posting on forums about their complaints and solutions or posting negative app reviews are your core audience and represent your audience as a whole, you are (most likely) wrong, and that is one of the easiest ways to lose sight of your product. Listening to customer feedback is part of the feedback diet (with another key piece being data), but devoting too much to the customer feedback portion will leave your product malnourished.
Let’s take a look at this through the vertical of video games.
It’s a vertical that I find easiest to demonstrate the concept of the vocal minority. Like other areas of entertainment, such as movies and comic books, there are users who flood the comment section with their thoughts and feelings, and for the most part, spend more time discussing or complaining about video games and comic book then of actually playing or reading them.
I came across some interesting posts from forums in regards to games by Blizzard that help to demonstrate customer feedback and the vocal minority.
Overwatch Topic — “You are the vocal minority”
It links to a video from one of the team members at Blizzard discussing recent updates with their game Overwatch. He mentions a key point where players that were happy with changes are less vocal than those that are not happy with changes. Those that are happy with the changes are less likely to go to the forums to give their positive feedback. Instead, it’s the ones that have issues with the changes that are making their voices heard. Here is a link to the video with the timestamp to that moment.
He brings up the point that they take feedback from various sources: They look at the forums and customer feedback, but they are also looking at the data they are collecting from the way players play the game, and make changes moving forward.
World of Warcraft Topic -Why Blizzard Shouldn’t Listen to the Fans (or, Vocal Minority)
Here is another post I came across from a forum user in regards to World of Warcraft which shows the dangers of listening to the vocal minority:
The problem is average users are giving solutions (simply increase damage of ability x by z%) instead of telling why. You can see it in almost every “helpful suggestions” on this and every other forum. All have their ideas on what the devs should change but they never explain why. Or in case of generic UI design test users might say “move that button there” which may or may not be the right solution.
Instead designers are looking for the underlying reasons what went wrong. In case of “increase ability x” the truth might be somebody is exclusively doing PvP and never participates in PvE and they aren’t seeing the big picture. If devs simply increase ability x it could severely destabilize PvE, unless they also see the person giving that feedback is only doing PvP and then they can start thinking if the class needs a buff in PvP or not.
Or the correct answer with a button might not be moving it two inches to the left, but instead higher up in the menu structure or even into the main UI players are seeing all the time into the micromenu.
That is why listening is bad, while looking at what people are actually doing when they’re having problems is the right way to solve design issues. That is why every single blue post on the subject is asking for “dont tell us what to change, instead tell us what is wrong and we’ll figure out a change for it”.
This harkens back to a post I link time and time again, from Ken Norton on listening to customers. Take a look at the section on the leading adopters.
I came across the topic of pandering to the fanbase on TVtropes when doing research for this post.
*WARNING: TVtropes is one of those sites that it’s very easy to end up down a rabbit hole when you were planning to spend a couple of minutes there.*
Here is a link for the curious to the sectioning on pandering to the fanbase.
Below is an excerpt of some of the key points on pandering to the fanbase.
Generally speaking, the more intensely devoted fans in a fandom are usually outnumbered by the casual fans, but the more devoted a fan becomes, the more active (and louder) they become in the fandom. So while a few million casual fans might enjoy an episode without ever making it widely known, a handful of devoted and occasionally unhinged fans are screaming on a web forum about how the show is now Ruined FOREVER, which can be seen and heard by everyone… including the people making the show. The producers may then start pandering to these voices exclusively, believing them to be the voice of everyone watching (which they will often claim to be) — but “everyone” in this case may in fact consist only of a handful of people, and what this minority wants and what the other, less noisy fans want can differ drastically.
This presents a major problem. The property can end up becoming a private club, accessible only to a select few. Excluding the casual fans means they’ll simply drift away to find something else to spend their time on, and raising the entry bar too high means you run the risk of locking out new fans who may have possibly been interested in the property, but now find it too difficult to access.While the vocal minority might now be satisfied (and you can’t even count on that), they rarely translate to enough ratings and / or sales to justify the property’s continued existence — and to make matters worse, even this hardcore minority may begin to drift away for numerous reasons (changing tastes, burnout, lessened interest, etc). This results in diminishing returns ending in eventual cancellation if unchecked.
The excerpt above can be applied to your product.
While doing research for this post, I came across this great article from Basecamp:
Why I don’t worry about our app’s negative reviews
Let’s look at some key points.
No matter how many negatives reviews we get, there are a lot of people who really like our app — they just don’t write many reviews. It’s important to remember those happy, quiet customers and not get too hung up on negative reviews.
Here’s the thing — as much as you want them to, there’s really no strong incentive for someone to leave you a positive review.
For many people, your app is doing its job dutifully and everything is working great. It’s not changing their lives or revolutionizing their world, but it’s helping them get something done. They’re thankful it exists.
But for them writing a review is never going to be a priority. Even if they love your app and are raving to their friends and co-workers about it, giving you written, positive feedback is never going to compete against the hundred other things they’ve got going on in their lives.
I’ve worked at a company that fell too much into the customer feedback trap. I’m sure we all have.
Hindsight being 20/20, I could have done more to get the company to focus more on the data and the happy customers, as opposed to the vocal minority that were constantly sending in feature requests and asking for solutions and not telling us about problems. We were spending too much time trying to appease a small minority, which took its toll on morale and focus of the products we built.
Now, customer feedback is important, but only a part of the equation, the other part being data.
I’ve said several times throughout my posts that there is a large difference in between what users says vs. what they mean, or what they do. That is why you need to be able to collect data, understand the data (and not get distracted), and derive context from the data. This can help you find out what users are doing, and where their pain points lie.
A healthy feedback diet includes looking at both the data, and customer feedback, whether through reviews, forum posts, support tickets etc.
This balance will vary on the product, and whatever you do, you do not want to skew too much on one side. As I’ve discussed in this post, listening too much to customer feedback without looking at the data may lead you to listening to closely to the vocal minority, which can clash with the rest of your users. Bury yourself too deep into data and ignoring customer feedback and you end up down the data hole trying to make correlations that aren’t there, and miss some very obvious things.
It’s a tough balance. Try it out with your product and team until you can find a good balance.
Like what you just read? You can join the PMPaul mailing list here.
Originally published at www.pmpaul.com on June 23, 2017.
