Video Game Reviews Are Broken
Publishers want control — fans want entertainment
Video games are expensive — and it’s hard to tell if one will be any good before it comes out. Games aren’t movies or books, with their low cost-of-entry and their low-price point. If you spend $10 on a bad film, you’re out the cash and two hours of your time. If you complain to the management, you might even get your money back.
Blow $60 on a bad or broken game and there’s little recourse. Most retailers — both digital and brick-and-mortar — don’t take returns on video games. That’s changing, especially in the digital realm, but for most video game fans the problem remains. Pay for a game that sucks … and you’re stuck with it.
And the game’s industry releases a lot of broken garbage. No Man’s Sky hyped the world with promises of an infinitely explorable galaxy then delivered a buggy mess that didn’t do half of what fans expected. Aliens: Colonial Marines looked nothing like it’s pre-release trailers. Batman: Arkham Knight was unplayable for many.
That’s why video game reviews are so important. Metacritic, games journalists and the independent YouTube crowd help consumers to make informed decisions before they shell out big bucks for a bad product.
But video game reviews are broken. High-profile scandals in the mainstream games journalism de-legitimized the press in the eyes of fans and collusion — both real and imagined — between publishers and gaming sites sowed distrust among consumers. Worse, publishers constantly seek new and creative ways to sell as many copies of a game before critics can warn fans.
Then there’s the fans. Many people who play video games — myself include — pre-buy games based on highly-controlled and polished preview footage or buy new games before reading reviews. Then, when we’re burned, we blame journalists for reporting the hype and accuse them of working with the games industry to sell us broken products.
We all need to change.
On Oct. 25, 2016, Bethesda — the publisher and developer behind the Skyrim, Fallout 4 and DOOM — announced it was getting out of the games review business. “We want everyone, including those in the media, to experience our games at the same time,” Global Content Lead Gary Steinman wrote on the company’s website.
“We also understand that some of you want to read reviews before you make your decision, and if that’s the case we encourage you to wait for your favorite reviewers to share their thoughts.”
This was bullshit. Three days later, Bethesda released Skyrim: Special Edition a relaunch of the successful Skyrim. It sent review copies to mainstream outlets the day before the game’s release, but was perfectly happy to pepper Twitch and YouTube streamers with copies of the game upward of a week before release.
Those aren’t the actions of a company that wants “everyone, including those in the media, to experience our games at the same time.” Those are the actions of a company that wants as many people to buy its product as possible before more critical eyes get hold of it and warn consumers about potential pitfalls.
Thankfully, Skyrim: Special Edition is a good game. It’s a prettier Skyrim with some nice additions and the ability to mod the game on consoles. Bethesda didn’t need to hold back review copies from major outlets … but it didn’t hurt them to do so.
Bethesda’s decision to court streamers instead of traditional media is the latest in a trend within the industry. Publishers want to control the story of their game releases, suppress negative coverage and sell as many copies as possible critical reviews come out.
Video games are a pain to review and the people who do the job often rely on publishers to make that job easier.
Players can finish off small games in a few hours but massive, open-world titles such as Skyrim or No Man’s Sky can take dozens of hours before a reviewer can say if it’s good or not. Technical issues may crop up in the middle of a 20-hour experience. A game’s narrative might strain and break in the final moments of a 50-hour experience.
Typically — though not always — publishers send out review copies of games weeks in advance. This gives reviewers time to play through the game, figure out its quirks and deliver honest criticism.
Journalists try to time the release of a review with the release of a game. Players are hungry for information when a game drops and eager to learn if they should spend their money. Media outlets tend to get the biggest traffic on a game review during the week a game’s released so those review copies are important to both publisher and the media outlet. It’s a symbiotic relationship.
But more and more, publishers are pulling away. Bethesda began this new policy before it announced it. Earlier this year, it held back review copies of DOOM until the day before release and game’s critics scrambled to get timely reviews out the door while readers were still interested.
“The biggest problem I had with the new DOOM game was how the optional items take up a lot of down time with a lot of uneventful exploration,” George Weidman told me. Weidman critiques games on his popular YouTube channel Super Bunnhyop.
Weidman’s videos are long and cerebral. A recent video about The Witcher 3: Hearts of Stone focused on the game’s roots in Polish myth and pop culture. He doesn’t rush his critiques out the door, he takes his time and sometimes releases a video months after a game debuted.
“When I was not rushing through [DOOM] I was finding problems everyone else was glossing over. There’s a correlation, I think, between how much of a fair assessment you can make over how a game plays in the long term and how much time you can play it.”
An enthusiast press
Bethesda’s restriction of review copies is just the latest in a long line of publisher-led shenanigans aimed at controlling the press. To be clear, publishers are under no obligation to play ball with journalists and send out review copies.
For an outlet that routinely publishes negative reviews or writes stories the publisher would rather not see the light of day, why bother? Kotaku knows how it feels. The popular gaming site hasn’t had a relationship with Bethesda for years now.
A little more than a year ago, Kotaku editor-in-chief Stephen Totilo opened up about Kotaku’s combative relationship with publishers Ubisoft and Bethesda. In the article, A Price of Games Journalism, Totilo explained how the publishers blacklisted Kotaku in retaliation for the site covering leaks and tumult at the companies.
A year later and Bethesda isn’t giving out review copies to any outlets until the day before release. It looks as if Bethesda extended its blackout of Kotaku to every other major media outlet. When Totilo spoke out against the ban, some fans speculated that Bethesda and Ubisoft weren’t happy with his site giving out negative reviews. Now that Bethesda’s cut off everyone, that speculation has returned.
“A lot of people, in talking about the blacklisting, there point of reference is publisher’s cracking down on outlets after they get upset about a given review,” Totilo told me in a conversation we had last year after he’d published his explainer.
“No publisher, to my knowledge, has ever come after us at Kotaku for any of the reviews that we’ve written. So, to their credit, that’s been a non-factor. And we’ve written some harsh reviews and never heard a peep of complaint. Which is as it should be. I always found gratifying yet recognize that smaller outlets had suffered some of that exact pressure of complaints.”
“I also realize that we have some insulation by being part of Gawker Media which is diversified enough in terms of what it covers that it’s not dependent on video game advertisers. So video game advertisers couldn’t exert as direct a pressure on editorial as they potentially could at other gaming specific outlets.”
Which has happened before. Sites that write about video games tend to advertise video games. As of this writing, IGN — one of the highest viewed video game sites on the internet — is advertising Sony’s upcoming The Last Guardian. It’s not unreasonable to assume that a site which runs negative coverage of a game that’s purchased ad space might face repercussions.
That’s what happened at Gamespot in 2007 when it ran a negative review of Eidos Interactive’s Kane & Lynch: Dead Men. Gamespot staffer Jeff Gerstmann wrote a middling review of the mediocre action game and Gamespot upper management fired him for it.
Eidos Interactive had paid huge sums of money to advertise the Kane & Lynch on Gamespot and it wasn’t happy with the negative coverage. The publisher reached out to the site’s upper management and the suits caved to a publisher who had invested enough advertising dollars to exercise editorial control over the site’s reviews.
By most accounts, this was a one-off. A scary, frightening one-off, but still a rare occurrence. Video game fans had long suspected their favorite sites were shills for the games industry. Gerstmann’s firing and the confirmation, five years later, that it had been at the behest of a publisher fueled a call for better ethics in games journalism.
“At some point, starting in 2011, journalism became one of the most hated professions to go into,” Weidman explained. “I’ve always thought the accusations launched against old media were fairly thin and not necessarily funded with so much evidence.”
“As is the case with Gamespot firing [Gerstmann], you have the two companies enjoying a very beneficial relationship with each other in the years since, a very open dialogue cropping up, talking about how there was a team of very new, inexperienced PR employees who drove the efforts to fire him.”
Weidman also explained that, at the time, journalist took to the streets to point out the injustice. “There were fellow journalists who were picketing the Gamespot officers during the days when that was going down,” he explained.
“You really haven’t seen that since, before or after. You’ve seen much more anger from a lot of fans but you haven’t seen the same degree of culpability in the past 10 years even as this anger has boiled over at a much greater level than it did [then].”
In some ways, these ethical gaffs are understandable. Games media is as new as the industry itself and it grew out of the hobbyist press. The pioneers of games journalism tended to be fans first … and journalists second.
“The gaming press comes from an enthusiast background,” Totilo explained. “There’s not a lot of journalists in the games journalism world that have journalistic training. You also have a lot of nebulous distinctions between critics and reporters. A lot of people — myself included — who are in the gaming media do both. So we’re constantly living in both the worlds of reporting facts and giving opinion.”
Totilo told me he pushes Kotaku to report the news with transparency. “But the gaming press does come from a background of amateurism and a love of video games,” he said. “So I think that … as a result of that, it hasn’t pushed the gaming companies as hard as you might expect.”
He also believes the old media-press is getting better at it’s job, but feels most outlets still have one big hang-up — they cover the industry like a trade journal. “There’s an inordinate focus in the gaming press on sales figures, business performance … a lot of the gaming press, even if you read a lot of the so-called ‘main gaming websites’ and outlets are telling you a lot about how a company’s doing and less about what’s going on in gaming culture.”
That’s true, the press surrounding this year’s Call of Duty: Infinite Warfare focused on the game’s poor sales, not its gameplay changes, space-navy story and fan response.
This focus on the business end tends to enamor sites and writers to the P.R. marketing apparatus of game’s companies. E3, DICE and GDC are gaming conventions covered in lavish detail by every major outlet.
They’re also trade shows where companies come to peddle unfinished and unreleased games. It’s about engaging fans in the hype cycle and getting them to spend their cash on an unfinished product.
“They’re engineered to generate hype in advance of a release to maximize sales on day one,” Totilo said.
“So they reach out to tell [Kotaku] about their games when they’re ready to announce them six months prior to release. They would invite us to press conferences to talk to the game designers for ten minutes and play the game for five so that we could continue to generate copy that would get people to know about a game that they would buy sometime in the future.”
“A gaming press that falls in line with that and just covers games on the hype cycle and abandons them after they come out is not really doing interesting games journalism as far as I’m concerned. That’s how it goes.”
Cocktails delivered while you play
Controlling access to review copies and focusing outlets on the hype cycle aren’t they only methods publishers use to control coverage. The other is a weird and insidious practice known as the review event. Instead of sending out copies of games, some companies bring writers to the game.
This happened most recently with Komani’s Metal Gear Solid V. “They shipped journalists out to a Konami event and had them stay for several days in a Konami controlled facility where they played their Konami game,” Weidman said.
“They had less than four days to play this thing then they’re not going to play it as intended. Which, I think, is a huge factor influencing how reviews are written. How much time does a writer have to make their review? If they didn’t have the time to really get into the … second half of this game then they’re not gonna know what it’s like for consumers who make it that far. I really think the content of their reviews signposts that that happened.”
Metal Gear Solid V is a massive game. The base story alone takes around 40 hours to complete and dedicated players can rack up 100 more hours in side quests. No sane reviewer could have a clear picture of the game after Konami’s tightly controlled sideshow.
“For fear of spoilers, Konami invited journalists to review the game at five-day ‘boot camps’ tied to strict NDAs. We played between nine A.M. to five P.M., with no unsupervised play outside these hours,” Dan Dawkins of GamesRadar explained in his initial write-up of the game.
“That’s a maximum play time of 40 hours, assuming no stoppages for eating, drinking, stretching … or reality. So you’re trying to complete a 35-to-50 hour game … that you’ve been anticipating for five years, in a realistic window of 30 to 35 hours.”
“On one hand, you’re finally immersed in one of the deepest, most experimental, open-worlds in history — overwhelmed by side-missions, upgrades and secrets — on the other, haunted by a tick-tock race to reach the ‘end’ without knowing when that is.”
In the weeks after the game’s release, journalists and commentators figured out that Konami had snipped off the last act of the game. Tech-savvy fans dug through the game’s source code and discovered evidence of an abandoned story line that seems to flesh out a game that seemed to end early.
By the time the community discovered the cut content, millions had already shelled out $60 bucks to see a sequel they’d waited years for.
“[Review events are] the exception rather than the norm,” Totilo told me. “Maybe a handful titles have that per year. They tend to be the bigger games.” He said that his biggest problem with the practice is the tendency for games to change radically after release.
“These days so many games are patched when they come out,” he said. “You’re potentially playing a game that’s more buggy than what your readers or viewers will be playing. Or you’re playing multiplayer modes of the game before the game’s really been tested in the wild with thousands or millions of players playing it.”
“So you’ll see games that get positive reviews then they come out and suddenly the multiplayer infrastructure collapses or they the map design is imbalanced in such a way that you’d never see it before release but it’s so obvious when it comes out that you then ask a valid question about whether it was premature to review that game before it came out.”
It’s all meta
Another method publishers used to control both their developers and the press is weirder and, thankfully, a dying practice. Most — though not all — video game reviews come with some kind of score that helps consumers get a quick read on whether a new game sucks.
Sites such as IGN use a 10-point scale, others prefer a letter grade and a few even score a game out of a possible 100. Popular media site Metacritic — the Rotten Tomatoes of video games — takes those scores and averages them into a aggregate score.
Weirdly, it translates every numbered and lettered score into it’s own 100-point system. It uses an arcane system to aggregate the scores and it doesn’t update scores … even if the outlet it’s quoting does.
This is a problem for two reasons. The first is, as Totilo pointed out, games change. Bethesda’s Dishonored 2 on P.C. was unplayable for some gamers on release. It’s gotten better after a few patches and it’ll continue to improve. Some outlets will update their score, but Metacritic won’t.
Last year’s Rainbow Six: Siege is a very different game now than it was at release. Ubisoft patched, updated and added content to the game. Does it still deserve the aggregate score of 73? No. The game now is vastly improved from the game released, but casual consumers who hop onto Metacritic for a quick take won’t know that.
Worse, some companies tie bonus payments for developers to Metacritic averages. This famously happened in 2010 when developer Obsidian released Fallout: New Vegas and missed earning a bonus for all its hard work by one Metacritic point. Years later, New Vegas is the most beloved of the modern Fallout games.
The practice of tying review scores to bonuses started at Warner Bros. Games in 2004 when then-senior vice president and general manager Jason Hall tied pay-outs to review scores. The practice has died off in recent years, but the memory of it still lingers in the minds of fans.
“It’s a putrid way to treat your employees,” Weidman said, but he still likes Metacritic. “It’s more the world around Metacritic that misuses or misinterprets it. I also think Metacritic’s standards for including reviews in their score averages are wonky … I don’t like that they can’t update scores in their averages.”
“I actually think I value the Metacritic average of a score extremely highly when buying a game for myself. The first will be word of mouth but the second will be if something comes out to abnormally high reviews. I’m a helluva lot more interested. I still trust the system to some extent.”
“An easy, handy, list of reviews that average into a general consensus of a new release is a handy tool for a consumer. I’ve used it myself for basing my purchasing decisions for video games with.”
Weidman’s doesn’t score his reviews of games on YouTube so his reviews aren’t part of the Metacritic ecosystem. Sometimes, it seems, the only way to win is not to play. Totilo said he agrees.
“[Kotaku] never scored reviews,” he told me. “We’ve never done it … it’s such an imperfect thing to equate quality to a number. It’s just not something I was interested in and I felt that the more you compel people to read your explanation of what you think of a game the better it is for everybody’s appreciation and understanding of the game.”
The real ethics problems
This swirl of retaliation from publishers against low scoring games, bonuses tied to review scores, P.R. department focus on the hype cycle and string of busted games has fans feeling … conspiratorial. Which makes sense.
Look at No Man’s Sky. The Sony-backed space-exploration sim spent years in development — and years in the spotlight. Mainstream publications such as Wired, GQ and The New Yorker wrote slavish praise of the game despite never having played it. Stephen Colbert interviewed lead developer Sean Murray on The Late Show.
The game was a massive disappointment. Fans sought refunds to mixed results and decried a games industry that broke its promises and a games media that supported the lie by publishing less than skeptical coverage of the hype cycle.
Worse, Sony was paying for ad space on dozens of the same websites that were running the positive coverage in the months before release. From the fan perspective, it’s easy to see a world where the games media and games companies collude to sucker fans into buying $60 games that aren’t quite what the developer’s promised. Reviewers who don’t step out of line, get fired or lose access.
The truth, of course, is more complicated. Publications chase the hype cycle because the fans love to read it and companies love to talk about it. Provable stories of publisher retaliation for negative coverage are extremely rare.
Games are punishing to make and development often occurs right up to the release date so it’s sometimes hard to get early, previewable code to media outlets and there’s no guarantee — for the publishers — of coverage that translates into sales.
Increasingly, publishers are turning to a source of news and information they believe they can control and fans seem to trust more — independent new media personalities on YouTube and Twitch.
“And yet new media channels are genuinely more guilty of committing the ethical gaffs.” Weidman told me. “During the launch of Shadow of Mordor, you had a scandal develop where multiple channels accepted a publishing deal that had them give the game favorable coverage during its launch weeks. Whereas you didn’t see that happen in the old media.”
He’s right. In the weeks after the release of Middle-Earth: Shadow of Mordor, rumors circulated that WB was giving over early copies of the game to YouTubers who promised to give the game favorable coverage. In a filing with the Federal Trade Commission in the summer of 2016, WB confirmed it had. It’s a common practice that’s been going on for years and both new and old media have sounded the alarm.
Remember … Bethesda told the world it wouldn’t send out review copies of Skyrim: Special Edition to old media outlets the day before the game’s release and sent popular YouTuber’s the game a week early. To be clear, I have no proof nor reason to believe that Bethesda paid for favorable coverage of Skyrim: Special Edition.
It’s far more likely that Bethesda sent copies of Skryim: SE to YouTubers it knew already liked Skyrim. The streamers get a copy of the game they already want, Bethesda gets positive coverage and fans hungry for more get to watch some fun videos.
For Bethesda, it makes business sense.
Consumers need honest and entertaining games coverage
For Weidman, a guy who makes part of his living critiquing video games on YouTube, there’s two reasons to write and read video game reviews. “The first one, the most noble one, the easy answer, is that it’s to inform consumers regarding their purchasing decisions,” he said.
“I think the second one is almost more important. It’s why I read reviews, why I’ve always enjoyed reading reviews and why I wanted to get into it. That is to create entertaining reading material.”
That’s where new media is kicking old media’s ass. Gamers — especially younger gamers — tend to get their game info from videos on Twitch and YouTube rather than from traditional outlets. Part of that is that people don’t trust journalists anymore — a problem not unique to games journalists. The other part is that a video is inherently more entertaining and easier to consume.
It makes sense. Games are a visual and interactive medium and it’s easier to get a sense of a game by watching someone play it than it is to read 1,000 words about someone playing it.
“A commentated ‘let’s-play’ won’t necessarily be titled Review: The Video Game but it’ll function as an extremely long-form criticism of a piece of media that informs the public about something they want to buy while at the same time giving them some interesting media to consume,” Weidman said.
“Remember what it was like going to school on the days the teacher would play a video instead of stand in front of the class and lecture?”
“People always looked forward to that and not because they relished the opportunity to learn more in a shorter amount of time. There’s just a natural appeal to the medium.”
To Weidman, the problem is simple and the solution is difficult. Old media needs to be less boring. “Right now [it’s] getting [its] asses kicked by much less reputable organizations who are publishing much less accurate information who are getting away with it because they’re more fun.”
“I think it’s a knack a lot of modern-day journalists have lost,” he said. “I remember reading about Edward R. Murrow and seeing some original broadcasts in which he cut to footage to McCarthy bumbling drunk in an interview and not necessarily editing it, but cutting back to a snide, sarcastic comment from a journalist delivered in a stiff, 1950s fashion. But it was still actually executed with a lot more grace and charisma than you see in a lot of media.”
Murrow managed to balance objective reporting with entertaining coverage. Somehow, he managed to have it both ways. The former broadcaster was a legend and we’re just talking about video games here, but it’s still an important lesson.
YouTubers such as Jim Sterling, Pewdiepie and boogie2988 do better than a lot of old media outlets because they’re more fun to watch. For that matter, written-word outlets Kotaku, Giant Bomb and Rock, Paper, Shotgun do well for the exact same reason.
But there’s another big problem. Another huge part of the broken video game review problem fans are uncomfortable talking about — us.
Publishers keep doing this stuff because it works
Sony sold millions of copies of No Man’s Sky before anyone outside of developer Hello Games had even touched the thing. Studios push pre-orders and for years, gamers have given over their cash for games sight-unseen.
There’ll never be a shortage of the new Madden game, but EA convinces people every year that they need to pre-buy it to secure their copy.
“It’s in the publisher’s best interest,” Weidman explained. “The economics show that they make the most money and take the least risk by having the most control over media. For consumers, the economic rationale works the opposite. Don’t buy stuff until you’ve thoroughly researched to make sure it’s a good deal.”
But we don’t. Time and time again we fall for the publisher’s hyped spin, pre-order games we shouldn’t and complain on the internet when the game isn’t everything we thought it would be. Publishers exercise control over the games media to suppress information as best they can in the weeks leading up to and the weeks after a game’s release.
“I don’t know why people buy bad video games the day they come out and think they’re gonna be great or why people buy every big video game … day it comes out,” Weidman said.
“I feel the reasons people do this are more rooted in how we buy and sell and how we understand the marketplace. It’s not about how much control publishers have over games media.”
“I don’t know if it’s a human thing, so much as a human’s who grew up in a society where buying stuff makes you feel good.”
He’s right. The problem is the consumer. The problem is us. Sure, the publishers are taking advantage, but we let them. Every time a shitty, broken game comes out and we buy it sight unseen, it tells publishers that the shady tactics of delaying review copies and buying off YouTubers works.
Things will change. The advertorial loophole of YouTubers will close. Publisher EA, perhaps anticipating possible legal changes, is asking German content makers to disclose when EA pays them. Old media outlets are slowly figuring out they need to be more entertaining while also telling the truth.
But the hardest change will be ours to make. I’m a fan. I love video games. When a shiny new one comes out, my instinct is always to rush out and buy it. I don’t always hold myself back … but I’m getting better.
Fans keep getting burned because because we keep sticking our hand in the fire. The publishers made that fire damn enticing, and the critics and journalists talk about how beautiful the fire is, but ultimately it’s our responsibility not to burn our own damned hand.
Writing is hard. Money is short. Support this reporter.
