“I can’t understand why reviewers rate this game so highly”: How Experts Differ From Amateurs in Video Game Reviews

Tiago Santos
Sep 19, 2019 · 4 min read

This post summarizes the CSCW 2019 paper “What’s in a Review: Discrepancies Between Expert and Amateur Reviews of Video Games on Metacritic” by Tiago Santos, Florian Lemmerich, Markus Strohmaier, and Denis Helic.

Which movie to watch or which video game to play are decisions for which we often rely on the opinions of the press and of our peers and friends.
However, if you are a Star Wars or an NBA fan, then you will have faced press (henceforth expert) opinions which conflict with or even contradict those of other consumers (henceforth amateurs) of the same movie or video game. Are these anecdotal examples merely isolated cases or samples of a broader trend? In other words, are expert opinions representative of or even needed at all by the amateurs?

Exemplary reviews of video games by experts and amateurs. One case where both agree and another where both disagree is shown.
Exemplary reviews of video games by experts and amateurs. One case where both agree and another where both disagree is shown.
Figure 1: Some video games elicit diverging opinions by experts and amateurs (left part), but there are also examples of the contrary (right part).

To address those questions, we study opinion differences between expert and amateur reviews of video games on the review aggregator website Metacritic. In video game ratings by experts and amateurs, we observe the former rate games most often 8-out-of-10 and the latter award rather extreme ratings on opposite ends of the scale:

Histogram depicting the unimodal expert rating distribution and the bimodal amateur rating distribution.
Histogram depicting the unimodal expert rating distribution and the bimodal amateur rating distribution.
Figure 2: The expert rating distribution features one prominent peak at rating 8, while amateurs rate games often a 10, 9 or 0.

As it appears there are substantial differences in review rating, we ask 1) do expert and amateur reviews also differ in other dimensions, such as temporal dynamics and text?, and 2) which such characteristics predict video game reception among amateurs in the short- and the long-term?

We find a series of differences in review production by experts and amateurs, of which we highlight a few.
First, we find that experts react more quickly and less extremely with respect to game release, when compared with amateurs.

Graphs depicting the relative frequency of grades experts and amateurs award per time window shortly after game release.
Graphs depicting the relative frequency of grades experts and amateurs award per time window shortly after game release.
Figure 3: Experts (on the left) award more balanced ratings more quickly than amateurs (on the right), as the latter gives more 10s and 0s in the first week after game release date.

Interestingly, amateurs rate games long after they have been released. In such cases, we also observe the ratings they award tend to increase as the years pass after game release.

Graphs depicting a) games per month passed between review and game release dates, and b) mean rating per year after release
Graphs depicting a) games per month passed between review and game release dates, and b) mean rating per year after release
Figure 4: Amateurs still review games years after release (left figure). Amateurs appear positively biased towards older games (right figure).

Could it be that amateurs yearn for the “good old days” in gaming? To address this question, we inspect review language, again juxtaposing expert with amateur reviews.
A topical analysis of review text reveals that experts focus more on game context and modalities, while amateurs on appealing concepts and experiences.
We also find both discuss gaming nostalgia, using words such as “old”, “classic”, “retro” and the word “nostalgia” itself. The usage by amateurs of such vocabulary increases over the years after game release.

Graph depicting usage of nostalgia-related words per year passed since game release date
Graph depicting usage of nostalgia-related words per year passed since game release date
Figure 5: The usage of nostalgia-related vocabulary in reviews by amateurs increases over time.

Leveraging these insights for predictions of the short- and long-term reception of video games among amateurs, we get the best prediction performance with models based on reviews by both experts and amateurs. We find amateur ratings are highly predictive in the short-term horizon, while expert ratings are very predictive in the long-term.

Graph depicting the performance of a set of models predicting short- and long-term video game reception among amateurs.
Graph depicting the performance of a set of models predicting short- and long-term video game reception among amateurs.
Figure 6: Expert and amateur rating-based predictions of short-term game reception among amateurs work remarkably well (left figure). Long-term predictions of game reception among amateurs are notably harder (right figure).

Overall, our findings indicate substantial differences in rating and reviewing by experts and amateurs. Our work not only highlights challenges in bridging expert to amateur views, but also helps understand mechanisms both use to signal (dis)approval and (dis)agreement. We believe these findings may be leveraged to improve review aggregators, recommender systems and video game (and entertainment product) design, development and consumption processes. Finally, we contribute to the debate of wisdom of the crowd vs. the few, as we extend that discussion to the context of experience goods, in particular video games.


If you want more details on our work or if you find this research helpful to your own work, please consider reading and citing:

Tiago Santos, Florian Lemmerich, Markus Strohmaier, and Denis Helic. 2019. What’s in a Review: Discrepancies Between Expert and Amateur Reviews of Video Games on Metacritic. In CSCW.

Tiago Santos is a recipient of a DOC Fellowship of the Austrian Academy of Sciences at the Institute of Interactive Systems and Data Science of the Graz University of Technology.

ACM CSCW

Tiago Santos

Written by

PhD Student at the Graz University of Technology

ACM CSCW

ACM CSCW

Research from the ACM conference on computer-supported cooperative work and social computing

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade