Hi John —
There is data in those reviews, to be sure. But it may not be great for stats.
I used to work at TripAdvisor and a spin-off that wasn’t quite as successful called DigitalAdvisor — our trade and craft was ratings and reviews. We aggregated them from other sources and collected them from our users. We also did some studies to understand the meaning of reviews, in particular of the ubiquitous five-star system.
The most salient finding was that the “extreme review” phenomenon was directly related to effort involved. Rating systems that require only a single click got several orders of magnitude more data points, but when compared to those requiring even just a subject line the skew was stunning — more than 90% landing on one end or other. When users had to type something, the distribution was broader, and further the likelihood of an extreme was directly correlated to the number of words typed.
We tried an experiment where we threw out the ends counting only 2,3 and 4 and found that these ratings were a far better signal as predictor of quality. There were a number of other factors, especially time between first review opportunity and the review itself — lots of people were anxious to be heard even when the product was not for sale yet :-)
While these findings may vary for apps versus products or places, I bet the overall result is similar. I also bet that I have read as many reviews as you — they are a wonderful (if sometimes frightening) view into the human psyche!
Above all you’re right: while you may need to wade through a lot of angst, there are times that a turn of a phrase, or an observation shared can be tremendously englightening.
