How to Read Podcast Analytics Without Being Wrong

Much has been made of the coming roll out of Apple’s improved podcast analytics. It seems like we’ve beat the issue to death in this newsletter already, but it brings to mind another important, broad concept when it comes to data:

Don’t read into the data what the data does not tell you.

When it comes to podcasting, this is an especially easy mistake to make, because the data currently is so outdated in its presentation.

Of course, there are the usual culprits: Total downloads doubled when we doubled our episode output! Well, duh. That’s not a good thing.

But there are subtler mistakes that can slip into a reading of podcast analytics. Let’s explore them:

“This episode got 25% more downloads than our average episode!”

Great. Now, let’s dig into what that means.

How about a multiple choice question to illustrate. Does the above data conclusively mean that:

1. The topic of that episode was what your audience wanted
2. The title made them want to click it
3. Your audience wants more episodes like that

What did you guess?

The correct answer is b.

This doesn’t mean that both a and c cannot be correct as well, but the only thing we know conclusively is that your audience clicked play on that episode title more than your normal episode title.

Let’s take another more involved case study:

Kast Media worked on the marketing for a blockbuster podcast which will remain nameless for obvious reasons. We did not produce the show. The podcast regularly would get about 150,000 downloads per episode.

Then, one episode, that number was suddenly cut in half for no apparent reason.

Scrambling for answers, we dug in and realized that, while the show would vary between weekly and bi-weekly releases, this was the first time we had waited a full 15 days in between episodes. The Podcasts app by Apple, if left to its factory settings, is set to auto-download new episodes of podcasts subscribed to. But if that podcast goes silent for 15 days, new episodes will not be auto-downloaded.

“So we made a mistake by allowing 15 days to pass, right?”

WRONG.

Waiting 15 days simply revealed to us what was really happening: Listeners weren’t listening. The episodes were being auto-downloaded but not listened to. We had marketed the crap out of that show, got those 150,000 listeners, and then a far too large segment of them had become disinterested.

We had a content problem, and the only way we could have found out about that content problem was through the painful drop in downloads we experienced.

In fact, one possible takeaway could be to make sure 15 day breaks are more frequent, so that we know what’s really going on.

Had we continued, our numbers would have remained high, maybe even for the rest of the season. But advertisers wouldn’t have seen good returns on their investments and so would not have renewed, and we would have remained blind to our content problems.

Instead, we were made aware of the issues early enough to be able to rectify them, and grow again.

Again, at first look, the takeaway would be: Don’t wait 15 days!

But in reality, the lesson is far different than that.

Effectively reading data, especially when it comes to podcast analytics, is tricky and requires a deep dive into what precisely the numbers mean and what they don’t. Avoid assumptions based on a surface reading. Instead, read the numbers carefully with an eye toward identifying only what is being told you explicitly in them. Anything else is just conjecture.

Kast Media publishes a bi-weekly newsletter about all things podcast. Check it out here: News.KastMedia.com