Metacritic and the Business of Games

Babies and Bathwater


There has been quite a bit of sturm und drang in the video game world over Metacritic and the role it plays in the industry. For those unfamiliar, Metacritic is a site that aggregates the review scores of many different review sites and aggregates them into a “metascore” (through some non-public method of weighting) that is meant to represent a consensus opinion on each game. In theory, it is using a wisdom-of-the-crowd method to determine how “good” a game is in a way that eliminates the biases and quirks on individual reviewers.

One of the criticisms Metacritic gets caught up in is the value of review scores themselves. How can you possibly represent the value of a videogame in a single number, given all of the things which affect that number? I won’t be addressing that here, as it applies far more to individual review sites’ decisions to use scores in the first place and philosophical questions about how to judge art. Also, people are free to ignore those numbers if they feel that such scores are useless.

I’m far more interested in Metacritic’s place in the business side of the videogame industry.

The videogame industry shares many structural similarities to the tabletop game industry. There are developers (the equivalent of tabletop designers) and publishers. Developers create the game and publishers pay the bills (though the timing of payments differs between the two industries). Publishers pay developers some combination of fixed amounts for meeting design milestones (in tabletop this would be getting the game contract-ready) plus royalties based on sales.

This is important because these developer-publisher contracts have more and more frequently been using a game’s Metacritic score as a basis for determining some amount of the developer’s final payments. Specifically, some bonus payments will be withheld unless the game achieves a specified minimum metascore.

This has caused a subset of the community to be angry at Metacritic. They feel that it unfair for developer’s livelihood to be controlled by something so opaque and arbitrary. They point out that review sites don’t all use a 0–100 system and that Metacritic’s conversion of, say, 3/5 stars to a score of 60% doesn’t represent the true opinion of the reviewer. The inability to know which reviewer’s scores will be weighted how is attacked as making it impossible for a developer to ensure that their hard work will be rewarded. The fact that a single reviewer’s giving a game a low score for what Metacritic-critics call incorrect criteria (such as messages conveyed by the game’s story or art) can push a game’s metascore below the contractual threshold is a common nightmare scenario raised.

I’ve simplified this a lot. If you want a fuller example of the criticisms of Metacritic, I would point you to the below video by Totalbiscuit (one of the most popular YouTube videogame reviewers), or to this post of his.

Contrary to popular opinion, I believe that Metacritic is one of the few bright spots in what has become a structurally dysfunctional industry. Unlike the current generation of videogamers, I am old enough to remember how the industry worked before Metacritic became a part of contracts.

I remember how developers and publishers would shrug off bad reviews by pointing to sales numbers. This was especially true with licensed games (I purchased many a bad Star Trek game in my youth) which sold on the premise of the IP and was especially effective in an era when reviews were only accessible through a magazine (note to youngsters: something printed on glossy paper with words printed in ink) that would arrive a month after the game had been released.

I will start with the following assumptions about things we collectively want to be true:

  1. When you purchase a game, it is a quality product (not buggy).
  2. When you purchase a game, it meets any objective claims made in its marketing materials.
  3. Developers and publishers who make quality games are rewarded for doing so.
  4. Developers and publishers who make poor-quality games are not rewarded for doing so.

I will also assume that I have a single magical power: to remove Metacritic (or any similar ratings-based system) scores from all contractual agreements in the videogame industry. This means I don’t have the power to change anything else, including consumer behavior, industry ethics, or laws.

To determine how anyone is likely to behave, you should look at the incentives they face. What are the current incentives in the videogame industry?

Publishers

Publishers care about profit. This is determined by how many copies of each game they can sell and how cheaply it can be made. This can be achieved either by selling a quality product or by aggressively marketing a game and selling as many copies as possible before anyone knows that it is a poor-quality product.

In the current environment, both strategies work. Too large a portion of consumers either buy a game the day it releases or, worse, pre-order the game. Publishers have exacerbated the latter by devoting a non-trivial amount of development resources towards creating pre-order incentives in the form of content that is inaccessible unless you pre-ordered the game.

You might think that publishers would see a greater profit in creating a reputation for quality which will lead to higher sales in the future. Perhaps this would make publishers reluctant to release a poor-quality game? History tells us no.

Publisher Sega released Alien: Colonial Marines in February of 2013. It was critically panned for both being poor on its own merits as well as for being drastically worse objectively versus the “gameplay” footage that had been shown in its marketing materials. Despite this, the game had sold 1.31 million copies in the US and Europe by the end of the next month. The game became a scandal, but the money had been made.

A mere 19 months later, Sega released Alien: Isolation. The same publisher, the same IP license, and the same pre-release hype as Colonial Marines all coming together again. Surely consumers would punish Sega for wasting their money in the prior year.

Nope. Within six months this game had sold over 2 million copies. Yes, it was a higher quality product and deserved its sales. However, this just goes to show that there is no downside for a publisher who knowingly lies about and releases a bad product. Consumers, especially the target demographic for videogames, have short memories.

This means that while currently #3 on our list of desired outcomes is true (it is possible for a publisher to make a profit on a quality game), the other three goals do not hold.

Would getting rid of Metacritic contracts improve the situation? No. These contracts are designed to affect developer behavior, not publishers. As most consumers are not aware of the contractual stipulations between publishers and developers of any specific game, this can’t affect consumer behavior. This means that publisher revenue will not be (directly) affected either way by removing Metacritic.

Developers

Developer who receive funding from publishers are put into a difficult situation. They are tasked with making a high-quality product for as little money as possible under a schedule determined by arbitrary events such as conference shows and holiday sales seasons having nothing to do with actual development.

The high rate of turnover at development studios is an indicator of this precarious situation and how little margin there is for spending a little extra time and money to fix bugs. There is a concept known as the project management triangle. It is commonly known in the simple form of: “Fast, Good, Cheap: Pick two.”

Before Metacritic became a part of publisher-developer contracts, developers had to meet two requirements in order to be paid: stay under budget and meet delivery milestones. In such an environment, it is easy to see which two side of the triangle would get prioritized.

Metacritic contracts change that. There is now an immediate financial reason for developers to care about the quality of a game. Before, any such incentives were fuzzy and speculative. Maybe a few more copies would be sold. Maybe they’d get another game funded next year. Maybe.

Now, there is an exact dollar figure and date attached to quality. Yes, the other two pressures of schedule and budget are still there. But Metacritic contracts mean there is finally something to compete with them.

But there are problems with Metacritic

The critic-critics are right. Metacritic’s algorithm is secret. It might not even be constant. Furthermore, the reviews that it uses as raw materials are themselves unobjective. Videogame reviewers are biased, unpredictable, and corruptible. It is frustrating to know that a single reviewer having a bad day can push a metascore from just above to just below the payment line.

But we shouldn’t let the perfect be the enemy of the good. Removing Metacritic (or some other form of review-based) clauses from publisher-developer contracts will have one effect: developers will have one less reason to care about delivering a quality product. Let’s go back to our list of goals:

  1. When you purchase a game, it is a quality product (not buggy).
  2. When you purchase a game, it meets any objective claims made in its marketing materials.
  3. Developers and publishers who make quality games are rewarded for doing so.
  4. Developers and publishers who make poor-quality games are not rewarded for doing so.

Review-based contract clauses incentivize all four of our goals. Nixing them removes that incentive. Sure, there may be other incentives to meet those goals, but why not use every tool in our bag. There is some, perhaps small but non-zero, downside to removing Metacritic’s effect on the videogame industry. What would the upside be?