“I can’t understand why reviewers rate this game so highly”: How Experts Differ From Amateurs in Video Game Reviews

This post summarizes the CSCW 2019 paper “What’s in a Review: Discrepancies Between Expert and Amateur Reviews of Video Games on Metacritic” by Tiago Santos, Florian Lemmerich, Markus Strohmaier, and Denis Helic.

Which movie to watch or which video game to play are decisions for which we often rely on the opinions of the press and of our peers and friends.
However, if you are a Star Wars or an NBA fan, then you will have faced press (henceforth expert) opinions which conflict with or even contradict those of other consumers (henceforth amateurs) of the same movie or video game. Are these anecdotal examples merely isolated cases or samples of a broader trend? In other words, are expert opinions representative of or even needed at all by the amateurs?

Figure 1: Some video games elicit diverging opinions by experts and amateurs (left part), but there are also examples of the contrary (right part).

To address those questions, we study opinion differences between expert and amateur reviews of video games on the review aggregator website Metacritic. In video game ratings by experts and amateurs, we observe the former rate games most often 8-out-of-10 and the latter award rather extreme ratings on opposite ends of the scale:

Figure 2: The expert rating distribution features one prominent peak at rating 8, while amateurs rate games often a 10, 9 or 0.

As it appears there are substantial differences in review rating, we ask 1) do expert and amateur reviews also differ in other dimensions, such as temporal dynamics and text?, and 2) which such characteristics predict video game reception among amateurs in the short- and the long-term?

We find a series of differences in review production by experts and amateurs, of which we highlight a few.
First, we find that experts react more quickly and less extremely with respect to game release, when compared with amateurs.

Figure 3: Experts (on the left) award more balanced ratings more quickly than amateurs (on the right), as the latter gives more 10s and 0s in the first week after game release date.

Interestingly, amateurs rate games long after they have been released. In such cases, we also observe the ratings they award tend to increase as the years pass after game release.

Figure 4: Amateurs still review games years after release (left figure). Amateurs appear positively biased towards older games (right figure).

Could it be that amateurs yearn for the “good old days” in gaming? To address this question, we inspect review language, again juxtaposing expert with amateur reviews.
A topical analysis of review text reveals that experts focus more on game context and modalities, while amateurs on appealing concepts and experiences.
We also find both discuss gaming nostalgia, using words such as “old”, “classic”, “retro” and the word “nostalgia” itself. The usage by amateurs of such vocabulary increases over the years after game release.

Figure 5: The usage of nostalgia-related vocabulary in reviews by amateurs increases over time.

Leveraging these insights for predictions of the short- and long-term reception of video games among amateurs, we get the best prediction performance with models based on reviews by both experts and amateurs. We find amateur ratings are highly predictive in the short-term horizon, while expert ratings are very predictive in the long-term.

Figure 6: Expert and amateur rating-based predictions of short-term game reception among amateurs work remarkably well (left figure). Long-term predictions of game reception among amateurs are notably harder (right figure).

Overall, our findings indicate substantial differences in rating and reviewing by experts and amateurs. Our work not only highlights challenges in bridging expert to amateur views, but also helps understand mechanisms both use to signal (dis)approval and (dis)agreement. We believe these findings may be leveraged to improve review aggregators, recommender systems and video game (and entertainment product) design, development and consumption processes. Finally, we contribute to the debate of wisdom of the crowd vs. the few, as we extend that discussion to the context of experience goods, in particular video games.

If you want more details on our work or if you find this research helpful to your own work, please consider reading and citing:

Tiago Santos, Florian Lemmerich, Markus Strohmaier, and Denis Helic. 2019. What’s in a Review: Discrepancies Between Expert and Amateur Reviews of Video Games on Metacritic. In CSCW.

Tiago Santos is a recipient of a DOC Fellowship of the Austrian Academy of Sciences at the Institute of Interactive Systems and Data Science of the Graz University of Technology.