Online Product Reviews Are Broken, And Fixing Them Won’t Be Easy
Search for almost anything on Amazon and your screen is flooded with beautiful product shots and 4.5-star reviews. Great, right? The e-commerce behemoth once again delivers high-quality products at reasonable prices thanks to the feedback of users that have combed through the chaff. But is that really the case?
Setting aside the rampant problem of fake reviews, I want to focus on three problems posed by legitimate, verified reviews: qualifications of the reviewer, difference in class of products, and the timing of reviews.
Grandma Gives It 5-Stars
Reviews come with a certain authority. Not only are they privileged enough to get a prominent place on the product page, but their mere publication has a sort of tacit endorsement by the brand that hosts them. “This is not inflammatory or false information,” we might infer from the mere existence of a review or rating online.
But online reviews and ratings are singular for the implicit trust we place in the opinion of their author, a stranger about whom we know literally nothing. We know so little, in fact, that we can’t even say with certainty that the person making these claims is even a person at all and not a robot paid to artificially inflate the positive stats.
Imagine that you were in the market for a new camera and you’d decided to finally go big and get a fancy, semi-professional DSLR. Now imagine that you were at the camera store, deciding between a Nikon and a Canon that were similar in price and ability, and you weren’t sure what sets either apart. Would you walk outside, grab some stranger and ask about which camera they would recommend? Of course not — because you couldn’t be sure they would have any authority on the matter.
So instead of asking some random stranger, you begin asking people you know in your daily life about which brand is better. An acquaintance at work tells you he bought the same Canon you’ve been eyeing, but says, “I love it man. It takes, like, real good pictures. Real good. It’s got all these buttons and stuff though, and I don’t even know what most of them do. One time I pushed the wrong one and all my pictures were all white and stuff, so I had to take it back to the store and they had to fix it. But I don’t think that was ‘cause of the camera.”
Would you trust this coworker’s sage advice? He gave it a gushing, 5-star review at first, but wasn’t really competent in the operation of the camera or its capabilities beyond taking pictures ‘real good.’
So finally, you ask a friend that’s practiced amateur photography for years, someone with a similar technical background and that has similar use cases as your own. Turns out that they started on Canon and switched to Nikon a few years later because the Nikons supported older lenses that could be picked up cheap on ebay whereas the Canons only accepted the more expensive, modern lenses. It’s a crucial point, and it sways you toward the Nikon.
When you don’t know who’s giving the review — what their competency and experience with the thing that you’re researching is — you’re rolling the dice and hoping that the person on the other end is the experienced friend and not the easily impressed coworker.
One Man’s 2-Star Is Another Man’s 7-Star
Different customer segments expect different things from the products they buy. If you’re a professional pianist, your review of a piano will echo the nuanced expectations you’ve developed through decades of sitting behind the keys, and the sort of detailed rigor you bring to that analysis simply cannot be learned any other way. It takes time and dedication, and while less experienced players may have their own opinions about a particular model, they will lack the same insight that a professional would offer.
But imagine that those same two groups review some digital keyboards. The professional has access to some seemingly astounding equipment — the finest the electronic instrument community has to offer. Nothing she tries retails for less than $2,000. These are professional instruments. She plays them all and reviews them with a wide array results, some 5-star, but one as low as 2-stars because the keys lacked the range of after-touch she expected and the sounds were subpar in the context of the other keyboards she’d been playing.
The second group of keyboard reviewers are taking an Introduction to Music Production class at the local community college and a representative from a local music store comes in with a variety of entry level synthesizers for them to try. Nothing is over $500. Every single keyboard they touch leaves their jaw on the ground. Sure, some sound better than others, but they didn’t even know synthesizers could sound this good at all. They’re living their dream, and would gladly take any one of these instruments. They give everything in the room 5-star ratings.
At first, this seems like the same problem I described before. We don’t know the qualification of the person reviewing the products, so we’re not sure who to believe. But the problem goes beyond that.
From the ratings alone, it’s difficult to know at face value if something is over-rated entry level gear or under-rated professional gear.
Is the 2-star keyboard objectively a 2-star keyboard if we look at every keyboard in the world? Or is it only 2-stars to musicians in the rarefied air of professional song making? The subtleties that irked the professional would be invisible to the untrained rookies that are floored every time they hear a square wave. They would rate the 2-star synth the same as every other instrument that day: 5-stars.
The counter-argument is obvious — the amateurs don’t need the professional gear, so if they’re shopping for their first instrument, it doesn’t matter if they pass by the 2-star instrument. But the problem arises for the huge number of hobbyists that sit somewhere in-between: the products are intended for different audiences, and subjective opinions about them that are informed by experience or lack thereof are presented as objective fact. How can we tell the difference?
In this case, there’s a real-world example I’ve seen recently. If you want to record sound into your computer using an external microphone, you’ll need an audio interface. They vary in price and quality, from consumer gear under $100 to professional recording studio rigs that cost in the tens-of-thousands of dollars. Here is a good semi-professional interface rated at only 3.6-stars despite having a modal rating of 5-stars. Here is a consumer-grade, entry-level interface that has a rating of 4.3-stars. Is the $60 2-channel interface really 20% better than the 30-channel $2,000 interface? Of course not. But at face value, the ratings present them as though it were the case.
It Works Today. Who Cares About Tomorrow?
How many charger cables have you gone through with your phone? How many of them got tweaked somehow, and after a pathetic period where you could wiggle the cable just the right way and balance it with the deft of an Alexander Calder sculpture to nurse a little juice out of it, it finally just quit working altogether?
Now how many of cables like that do you think have 4 or 5-star ratings?
All of them!
I get that cables are a semi-disposable resource and we shouldn’t expect them to survive the horrors of modern life to which we subject them, but the reason those cables have 5-star ratings and not 1-star ratings is because we ask people to review the product the day they got it and never ask them how it’s going 6 to 9 months down the road. Occasionally, you’ll find someone who’s updated a review when something goes wrong, but you never see someone come back to tell us when it’s gone right.
Of course the cable works the day you bought it. The fact that the manufacturer has managed to build it to a tolerance that can survive a journey in highly-engineered packaging and shipping containers is no great achievement. What is impressive is a cable that can survive being in the purse of a millennial that takes it between her car, house, boyfriend’s house, work, on airplanes, in hotels —building a cable that can survive all that is more impressive than landing a man on the moon if you ask me.
Of the three problems I’ve presented, this is clearly the most easily solved. We have to ask people how they feel about their purchase after the initial excitement of getting a package in the mail has subsided, when they’ve used the thing in the real world for a while.
I Want To Date Your Ratings and Reviews
The first two problems are not easily solved. Figuring out and verifying who is an authority on a subject is no small task, and solving the first problem will often exacerbate the second. No audio professional in their right mind is going to recommend buying a $60 audio interface from Behringer, but that doesn’t mean it isn’t right for some customers. From the number of 5-star reviews it has, we can see that’s clearly the case.
The thing is, I don’t always want reviews and ratings from professionals. I want to see how people like me felt about a product. I want to know what swayed them to purchase what they bought, and I want to know how they felt about that purchase six-months down the road. On topics where I’m an expert, a professional opinion is appropriate. But for topics where I’m not, I’d rather hear from someone who made the same choice I’m about to make, but a few months earlier.
And when I’m browsing for products on big e-tailers like Amazon, I want to be able to sort by “Highest-rated by people like me.”
And that’s where dating comes in.
The same matchmaking algorithms that help us find love in this topsy-turvy modern world could help us find record players with which to woo our would be lovers. They could direct us to interesting hotel rooms for weekend getaways or find mini-vans for newly formed families. They can connect us with the thoughts and opinions of people who, given similar conditions, have made the same choice at some recent time.
Much of the effort to socialize shopping has centered on connecting us with how our friends behave: what they buy, what they’ve reviewed, and so on. But while that may solve problems like restaurant recommendations, it limits the search space to only the products that my direct group of friends has had experience with. That’s not a big set for a lot of categories, particularly if it’s something neither I nor my friends have any prior involvement with.
So the matchmaking algorithms of online love may yet find a new application in connecting us with other people who’ve already faced the same paradox of choice that we face.
Because in the end, products are about the people who buy them, and online ratings and reviews are only useful if they come from the right people.
