Tom Harrison
Aug 26, 2017 · 2 min read

Hi John —

There is data in those reviews, to be sure. But it may not be great for stats.

I used to work at TripAdvisor and a spin-off that wasn’t quite as successful called DigitalAdvisor — our trade and craft was ratings and reviews. We aggregated them from other sources and collected them from our users. We also did some studies to understand the meaning of reviews, in particular of the ubiquitous five-star system.

The most salient finding was that the “extreme review” phenomenon was directly related to effort involved. Rating systems that require only a single click got several orders of magnitude more data points, but when compared to those requiring even just a subject line the skew was stunning — more than 90% landing on one end or other. When users had to type something, the distribution was broader, and further the likelihood of an extreme was directly correlated to the number of words typed.

We tried an experiment where we threw out the ends counting only 2,3 and 4 and found that these ratings were a far better signal as predictor of quality. There were a number of other factors, especially time between first review opportunity and the review itself — lots of people were anxious to be heard even when the product was not for sale yet :-)

While these findings may vary for apps versus products or places, I bet the overall result is similar. I also bet that I have read as many reviews as you — they are a wonderful (if sometimes frightening) view into the human psyche!

Above all you’re right: while you may need to wade through a lot of angst, there are times that a turn of a phrase, or an observation shared can be tremendously englightening.

)

    Tom Harrison

    Written by

    30 Years of Developing Software, 20 Years of Being a Parent, 10 Years of Being Old. (Effective: 2019)

    Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
    Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
    Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade