Putting “readable videos” to the test on Facebook

Since last June, I’ve been fascinated with “readable video”: the storytelling genre that sprung up in response to Facebook’s Harry Potter-inspired decision to autoplay video with the sound off.

In recent months, I’ve begun producing them myself and pitching them to clients as one of the most effective ways to reach a large audience on the web today.

But is that statement true? How effective are they really?

Here’s one data point to consider: If you look at the libraries of the top ten Facebook video publishers, you’ll notice that most of their videos are comprehensible with the sound off. Some of these are “readable” (with overlaid text) and some are just inherently visual (recipe or fashion tutorials, for example). Regardless, this type of content certainly isn’t getting in the way of massive reach.

But as someone specifically interested in explanatory storytelling around news and science, is readable video actually better? Does it lead Facebook viewers to watch more of this type of content than they would otherwise?

I didn’t have a concrete answer, so I set up a small experiment to figure it out.

First, a quick bit of background

If you’re unfamiliar with the “readable video” genre, here is one I produced earlier this spring:

When you think about the contexts in which many people browse Facebook …

  • In public (waiting in line, riding public transit, etc.)
  • At work (at one’s desk, eating lunch, etc.)
  • On the couch at home

… it makes sense that readable videos like this one would work well with users. If you’re browsing FB in public or at the office, consuming a video with sound can often be cumbersome or disruptive. If you share your home with others, the same sensitivities apply.

Moreover, the user experience is seamless. These videos start automatically as you scroll down your feed, you can sample them without hitting a play button and you can simply keep scrolling if you’re not interested.

Readable videos now come in a few different shapes and sizes. Some of them — like the coral bleaching piece above — feature lightly animated text that takes up a good amount of screen real estate. Some take the subtitle approach, always displaying the text across the bottom of the frame. And some publishers use narrated content and upload their subtitles to Facebook’s backend (via .srt files), which then appear like generic closed captions. Here’s a screenshot (from this Vox.com vid):

What works best?

To gauge whether readable videos really lead to more engagement, I created a short video that highlights the features of NASA’s Curiosity rover. I forked it into three versions, the first of which is narrated:

The second version is narrated, with closed captions (the captions would appear automatically in FB’s newsfeed, but in the embed below you have to turn them on via the settings menu):

The third version is fully readable, with no narration and big animated text throughout:

I then used Facebook’s ad platform to show the videos to users across the United States, between the ages of 18 and 65, with an interest in science and space.

In total, I drove approximately 7,000 impressions to each version. All three videos performed similarly when it came to getting viewers to watch at least three seconds — the threshold at which Facebook counts a “view”. The conversion rate was around 35% or approximately 2,500 views for each (this would likely be a bit higher if the content wasn’t coming in the form of an ad from an unfamiliar source).

Beyond the three-second mark, engagement started to diverge as the audience retention graph below shows (using the following milestones: 15% 25%, 50%, 75%, 95% and 100%):

As you can see, the two narrated versions track quite closely, while the readable version holds its audience significantly better, particularly up to the 50% and 75% marks.

Moreover, the readable video wins out when we look at the “average percent of video viewed” — another metric provided by Facebook:

That’s a boost of about 25% over the two narrated versions.

Why does attention time matter? Because if you’ve gone through the trouble of creating a video (no small feat), it’s because you have an important story to tell or message to communicate. The more of that story that gets absorbed by your audience, the better.

In addition, more attention time leads to increased shares and wider reach, as Upworthy discovered in its own data: “The propensity to share is closely tied to how much attention each user pays … It’s those who make it to the end who are the most likely to share.”

My test, while not perfect, suggests that if you want to increase your chances of high engagement on Facebook, you should go with a fully readable video.

I use the phrase “increase your chances” because this is not a silver bullet. A readable video is not going to guarantee a viral success. The real challenge is creating a piece of content that is genuinely engaging or useful and pulls the viewer in.

If you can meet that prerequisite, then using this particular medium will give you the best shot at attracting a big audience on Facebook.

While the results of this test are emboldening, they also raise a bunch of follow-up questions that I’d like to probe:

  • What variables have an effect on the initial conversion (from impression to “view”)?
  • If the video starts with a person speaking into the camera, does that change the results?
  • What happens if you replicate this test, but isolate for mobile and desktop users?
  • How does the pacing and density of the onscreen text effect engagement?
  • This test was performed with a video that is around 60 seconds long. Does readable video’s advantage hold at two minutes? Three minutes? Four minutes?
  • Do these results hold if you test the same variations on a page’s followers, rather than on strangers?

Any others that come to mind?