Actually, Maybe Rotten Tomatoes Scores Do Affect Box Office Performance?

Nearly two decades after Rotten Tomatoes burst onto the scene, it seems we’re finally reaching a boiling point in the impact of critical acclaim on Hollywood. Last week, Brooks Barnes in the NYTimes captured the fears of studio executives increasingly concerned about the impact the review aggregator has on box office performance. Yves Bergquist, the Director of the USC Entertainment Technology Center (editor’s note: fight on!) responded with a data-driven post arguing that no, Rotten Tomatoes scores do not have any correlation with box office performance. Seemingly every major outlet picked up the story, and that was that. Case closed, right?

I’m not so sure.

Missing The Big Picture

Let me say upfront that I’m no statistician. I feel like a genius every time I do a vLookup or pivot table in Excel; I certainly can’t calculate a basic Pearson Product-Moment Correlation Coefficient.

But looking at Mr. Bergquist’s analysis, I found myself more skeptical than convinced.

His data ignores arguably the most important consideration: budget (in particular marketing). Larger budgets reflect greater risk for Hollywood, and marketing always has an effect on box office. If you simply compare the new Pirates of the Caribbean ($172 million domestic gross; 29% Rotten Tomatoes score) with Get Out ($175 million / 99%), you might think that Rotten Tomatoes didn’t have an effect. But Pirates had a $230 million production budget, not including marketing (which can reach as high as $200 million alone for blockbusters); Get Out cost $4.5 million, with around a $25 million marketing budget. The film with a 29% score lost at least $57 million domestically; the one with a 99% netted around $140 million.

Furthermore, Mr. Bergquist considered every film that has grossed at least $1 million, but this has the effect of muddying the data. Hollywood is extremely top-heavy; the top-25 grossing films in 2016 represented over 50% of all box office gross, and the top-25 in 2017 represent over 61%. Right now — for better or worse — studios rely on big, expensive wins, and have for years. We should focus on what’s happening at the top.

So I decided to do a little brute force analysis of my own, focusing on the most successful and the most expensive films of the year — the ones Hollywood needs to get right.

Key Data

Let’s review a few assumptions in my numbers.

1. Rotten Tomatoes generates 42% of its traffic from the USA. China is responsible for a huge chunk of worldwide box office, but represents at most 2.5% of Rotten Tomatoes’ traffic. Because of this disparity, I’m only considering domestic box office, as this represents the strong plurality of Rotten Tomatoes visitors.
2. I’m focused on whether a movie is “Certified Fresh” (at least 60% of all reviews are positive) or “Rotten” (fewer than 60% of reviews are positive), as this is the core branding that Rotten Tomatoes applies to each film. How far above or below 60% a film scores is irrelevant for this purpose.
3. Marketing budgets are incredibly difficult to find, so I am using production budgets. Mr. Bergquist argues this is notoriously inaccurate, and that production budgets are a bad proxy for a marketing budget. He may be right, but a big budget movie will almost always find itself with a large marketing budget, and the research I’ve done on a few individual films suggests it’s not that far off. Regardless, the BoxOfficeMojo data is the best I could do at scale.
4. I am excluding small sample sizes or missing data. For example, Baahubali 2: The Conclusion scored a 100% on Rotten Tomatoes…on eight reviews. Meanwhile, I was unable to find a budget for The Glass Castle. These movies were unlikely to be relevant to the below analysis anyhow, but both were excluded.
5. You can view my source data here.

The Color Of Money

Let’s keep it simple and start by checking in on the top-25 grossing films in 2017:

That looks like a pretty solid trend to me: the top-10 grossing movies of the year are all Certified Fresh, and 19 of the top 25 (76%) are Fresh. Regardless of what drives box office receipts, we can at least say confidently that the movies that have risen to the top of the box office this year have tended to be well-reviewed.

Perhaps this correlation doesn’t exist when you add in the remaining 130 movies that have grossed over $1 million, but again, these 25 films represent 61% of the YTD box office.

Risky Business

Still, I don’t think the above data tells you a lot about the strength of Hollywood’s business model as it relates to Rotten Tomatoes. Movies that gross a lot tend to be good, but it’s a fallacy to look at gross revenue in a vacuum. Hollywood studios rely on big, pricy tentpole films, not diamonds in the rough such as Get Out. So let’s take a look at how the most expensive movies performed relative to their budgets and reviews.

Woof. There’s a lot of color there, I’ll break out the key findings:

1. Again, 19 of the 25 most expensive movies of the year (76%) had a correlation between Rotten Tomatoes “freshness” and domestic profitability (domestic gross vs budget). Nearly half of the top 25 are rated Rotten (fewer than 60% positive reviews).
2. The correlation is particularly strong for the Rotten movies. 11 of the 12 poorly-reviewed films generated a domestic net loss.
3. A handful of the Certified Fresh films turned out a loss as well, but these still fared better than their Rotten partners. Four of the five non-correlating Fresh films saw a 3%-13% loss from budget to box office. Aside from Boss Baby, no Rotten film’s box office came within 15% of reaching their budget, and only Power Rangers was even within 25%. The Rotten films generally saw a major loss domestically.
4. Overall, 16/25 films saw a domestic loss. That’s a terrible success rate, only sustainable thanks to the international market.

I’d love to know what percentage of total Hollywood studio production spending is represented by the above 25 films, but haven’t been able to find reliable data for that.

Nonetheless, these represent the biggest individual investments Hollywood has made in 2017 thus far. Nearly all of the films that received a Rotten rating lost money domestically, while nearly all of those that received a Fresh rating at least came close to earning back their production budgets. Overall, the Fresh-rated movies earned over $1.3 billion more than their budgets (their combined box office was 176% of their budgets). The Rotten-rated movies lost over $676 million from budget to box office (their box office was barely 60% of their total budgets).

It’s hard for me to believe that Rotten Tomatoes means nothing.

What’s Next?

The reality is, however, that most of those films likely did end up profitable once the international market came into account. The strength of the Chinese market in particular — which, as I noted above, does not seem to traffic Rotten Tomatoes — continues to prop up poorly reviewed films that the domestic audience is increasingly turning away from.

Perhaps this is sustainable; perhaps not. But if every Rotten film in the above list of the 25 most expensive movies of the year had performed as well relative to its budget as the Fresh films did, it would have represented a nearly $1.8 billion increase in net revenue. That’s a 23% increase in total domestic box office this year.

Mr. Bergquist did find a correlation between audience scores and critic scores in Rotten Tomatoes. He also found that there’s been less and less of a correlation between production budgets and box office over time. It’s possible that Rotten Tomatoes scores are not influencing audiences, but rather that audiences are already on the same page as the critics when it comes to blockbusters — they can tell from a trailer whether a movie is likely to be good or not, and the box office results and Rotten Tomatoes both reflect the poor quality of the film (rather than the score actually causing the box office result).

Whether the Rotten Tomatoes score influences the box office or simply reflects the mindset of the audience, the end result is the same: bad expensive movies perform poorly, and good expensive movies perform well.

The answer for studios is simple: shift their attention toward ensuring that their most expensive films are also among their best. And I see no evidence that it costs more to make a good film than a bad film.

So don’t complain about Rotten Tomatoes hurting your movie’s performance.

Just make better movies.