What do podcast ratings actually tell us?

We looked at all of them to find out

Though Apple won’t confirm it, prevailing podcast wisdom holds that the best way to bump a podcast up the charts is to get listeners to rate and review their show on iTunes. Certainly the number of ratings a show earns is a strong indicator of the number of people listening, but how useful is the rating itself — the number of stars (1–5) that listeners give the show? Based on our analysis, the answer is somewhat counterintuitive: The more ratings a show has, the less useful they are.

This is the first in a series of posts from Panoply that will draw lessons from a massive data set we’ve collected — essentially the entire universe of available podcasts. Over the coming months, we’ll analyze the data to explore broad trends in the industry that have up until now been invisible. Today: what can we learn from Apple’s star ratings?

A note on ratings vs. reviews: The rating refers to the number of stars, 1–5, given by a listener, while a review means a written assessment. Apple allows its audience to rate without reviewing, meaning podcasts generally have more ratings than actual reviews.

Very few podcasts have more than a handful of ratings

The first thing to jump out of the results is how few podcasts actually have a significant number of ratings. This isn’t terribly surprising given what we already know about podcasting as a whole: a few very large shows dominate the charts, but there’s a long tail of podcasts with small or even tiny audiences.

We collected data from all of the hundreds of thousands of podcasts in the US iTunes store as of September of this year, including the number of ratings and the average rating reported by iTunes. Out of all the podcasts in iTunes, roughly 80% have no ratings listed, meaning they’ve received fewer than five (iTunes will only publish the number of ratings for a podcast if the show has five or more, so we can’t look at lower numbers.) About 90% of podcasts have 10 or fewer ratings. In short, the vast majority of listener ratings are going to a very small number of podcasts. It’s an elite group — 0.1% of the podcasts listed by Apple — that have 12,000 or more ratings. These are the high-ranking podcasts we all know — the Joe Rogan Experiences, My Favorite Murders and This American Lifes of the world.

In general, lots of ratings = positive ratings

Ideally, the number of ratings would correlate with how many people have listened to a show, and the average rating would indicate how many of those listeners liked it. In theory, a lot of people might try a high-profile new show and hate it, leading to thousands of negative ratings and a low average number of stars. But this doesn’t show up in the data, for reasons I’ll discuss shortly.

The chart below shows a dot for every podcast, marking the number of ratings and star average. One thing to note: the vertical axis is scaled by powers of ten, meaning the scale jumps from 100 to 1,000 to 10,000. That’s because most podcasts are pretty small, but the big shows are truly enormous, getting tens of thousands of ratings. This lets us see all of them neatly on a chart together.

The big trend here: While shows with a small number of ratings run the gamut from an average rating of 1 to 5 stars, the higher the number of ratings, the more likely a show is to get close to a 5 star average.

With more ratings, the average rating is more positive.

But wait, I hear you saying, even the most popular shows must have a significant number of detractors. So what’s going on here?

Think for a minute about who actually leaves ratings. If you’re looking to buy a vacuum cleaner online, you’re liable to see online ratings either by people who think the vacuum is the greatest thing they’ve ever bought, or people who want the world to know they got ripped off. The people who feel ho-hum have less skin in the game and tend not to pipe up. In other words, the people motivated to rate are people with strong feelings.

But why do we only see strongly positive feelings here in the podcast listings? Odds are, the only people with strong feelings about podcasts are those who actively support them and listen to every episode. These super fans are most likely to post ratings. Where are all the haters? The difference may be the price point. Podcasts are free! No refund to demand. If a listener finds out a show’s not for them, they just walk away. If they already feel like the podcast was a waste of their time, they’re not going to take still more time to shout it from the rooftops — that is unless they have some other axe to grind.

The outliers help to make the case that this is what’s going on.

The outliers

Below I’ve picked out a few of these outliers: shows that have a solid number of ratings but less than a 3.5 average. What kinds of podcasts are these?

Some of the outliers are national news or political opinion shows: Bill O’Reilly, Fox News Flash, This Week with George Stephanopolous. These have average ratings roughly in the middle of the scale. And this makes sense: these are shows that have a high profile and that take positions many will disagree with. Here, the motivation to rate the show probably comes not from the quality of the show itself, but the strength of listeners’ political views.

Perhaps the two biggest outliers are I Am Rapaport and Unmuted. Both have a large number of ratings and both are hosted by well known people (actor Michael Rapaport and former sports columnist Jay Mariotti, respectively) who… let’s say, are strong personalities with plenty of detractors. A lot of the written reviews include sentiments I’d rather not repeat.

In general, a high average rating is basically a given for the vast majority of big podcasts. In a small number of cases though, the average likely reflects the degree to which a vocal portion of the internet really dislikes the host.

Still, star averages do have value for one important subset of podcasts — those medium-size podcasts with somewhere between 100 and a few thousand ratings. It’s here, in this middle ground between the vast majority of podcasts with few if any ratings and the rarified world of blockbuster shows, where listeners seem to be doing something valuable for the podcast community — giving their honest opinions about a program. Sure, even this subset may be tainted by companies that promise downloads or ratings for cash. But our advice to podcasters based on this analysis is to pay extra close attention to the first few hundred ratings and reviews you get. These may be from listeners who aren’t just jumping on a bandwagon, but who deserve your attention as you work to build an audience.