Determining “success” for online educational videos
Pre-teen girls and middle-aged men ended up watching the same educational video — this is why it should affect how we look at video as a learning experience.
The following video received over 25,000 hits on YouTube overnight on multiple occasions. Granted, the counts are modest in the context of most viral content, but it was noteworthy for us, a small educational media program at MIT. Of all the videos we’ve made (and that includes ones on the physics of skydiving and on humanoid robots), this one took off the most:
And not only has this video gotten more overnight hits than any of our other ones, its viewers, on average, are noticeably more diverse and watching a greater percentage of it (47% versus the 39% average for our channel).
The quick-and-dirty explanation of what happened:
- The topic (braces) personally resonated with a demographic that’s very active on YouTube (pre-teen girls and their mothers), and particularly active looking for videos on that topic
- It was promoted by someone/thing that already has a large — and very devoted — science/tech/engineering/math-inclined audience (MythBusters)
So, if you want to make an educational online STEM video that gets a respectable amount of views, or that retain audiences better, that’s one place to start. But if you want to use video in a learning experience, views and retention shouldn’t be the main things that matter, and the story of this particular video might explain why.
I once heard Logan Smalley, director of TED-Ed, say that we misunderstand the life of a web video by reducing its beginning, middle, and end to those literal parts. We think that a video begins when you hit ‘play’ and ends when the screen turns black. He argued that the life of web video was really much bigger, starting with the 140-character Tweet that announces its existence to the world and ending with all the likes, shares, comments, and YouTube rabbit-hole-ing that indefinitely occur. The video itself is just the middle. There was something interesting, or at least different, about the life of this video compared to the other 200 on our channel. Why?
The Beginning
In January 2015, I started a course called 20.219 (Becoming the Next Bill Nye: Writing and Hosting the Educational Show). Students would spend a month creating an online video on any STEM topic, with the following stipulations: it had to be under 5 minutes and they had to be on-screen. We had freshman, exchange students, grad students — all with little-to-no video experience — and Andrea, a fellow in the Sloan School of Business. Her first daily blog for the class reflected on the first assignment to record a “pitch” for a video. It was titled, ‘Andrea’s Struggles’:
The theme here is a) “we are experiencing technical difficulties,” and b) fear of the camera.
a) New camera, new tripod. Mac computer. Sigh. I’ll work on it again tonight.
b) Much bigger issue actually. This fear took several forms, starting with the thought that it’s all been done before. Then I read entrepreneur Seth Godin’s blog, which stated, “Originality is local… if we look hard enough we’ll find that turn of phrase or that app, somewhere else. But no one is asking you to be original. We’re asking you to be generous and brave and to matter… Sure, it’s been done before. But not by you. And not for us.”So I stopped being nervous about being original. Once I was able to do that I was able to discard my first idea — how to teach kids about system dynamics… Using the tenet of “talk about what you know” I did then come up with an idea.
So, my pitch, keeping my middle school audience in mind, settled on describing what happens to your teeth when you have braces. It’s a common enough problem with kids that age. Also, I recently worked for an orthodontic company that makes braces, so this is actually something I know a little bit about.
Goals for class: My primary goal is to become more comfortable with video as a format for expression of complex ideas. I’ve always loved science shows. In high school (in the 80s, remember I am OLD), my Latin teacher showed episodes of the original Connections once per week. I loved how it tied science and history together, which made science more approachable and “human.” … I think that is one of the primary roles of science education — to convey its “human-ness” and thereby its accessibility. I don’t think we will ever want to be taught by robots — there will always be a person somewhere to make it relatable.
In the pitch, Andrea continues to explain that her empathy for middle and high schoolers going through the process of braces motivated her decision to switch topics (she had just gotten hers removed). But even with the topic figured out, Day 2's blog post (again titled, ‘Andrea’s Struggles’) reflected the remaining challenges:
As for process, I managed to get the tech difficulties out of the way. Most of the barrier was psych — sometimes I feel too old to learn the new way of doing things (kids these days…). But I wouldn’t be here if I didn’t think I could learn deep down inside. Right? I also worry in the class that I’m a bit intimidating because I’m older. Also, “Sloan Fellows” sounds really “important” (and the program office has drilled into us to self-identify that way), but this isn’t really the right venue for that…
Over the course of the month, her idea went through several script revisions, workshops, rough cuts, screenings, and consultations with everyone from 12-year-olds to our class community, which included 5 instructors along with a varied student body. If you browse the stages of Andrea’s project over on the class blog, you’ll see that her idea begins as a facts-based “explainer” video. It presented new details on not-that-new, almost intuitive information — braces use forces to move your teeth around. Someone brought up that it felt a bit like an after-school special. But over time, her drafts reveal a small but crucial transformation into something that focuses on the non-obvious, almost profound notion that your own body (not the braces) breaks and rebuilds itself to straighten your teeth. As Chris Boebel, one of our instructors, would say — it makes the familiar unfamiliar. It takes a relatable, almost mundane occurrence and shows viewers a new way to look at it. It tries to elicit a response that’s not just, “I didn’t know that”, but “I didn’t know that, and I would have never expected that”.
Here is Andrea’s final project, the last iteration before we produced the Science Out Loud episode based on this video:
When we took over to produce the Science Out Loud version of her video, I think the biggest change was having coaches to direct and guide Andrea’s delivery during production. We also made some changes to her script, focusing on fleshing out the unfamiliar aspect of the familiar topic, and cleaned up some of the technical elements of production. After filming, she wrote:
Out of the entire experience, from ideation to script writing to shooting to editing to repeat, it would be easy for me to stick with the part I feel most comfortable with — the script writing part. But we only get growth when we throw ourselves at the things we suck at… I’m so glad I didn’t quit this class. But it was close there for a while…
While I can’t say I’m happy with my final product (I think I went a little crazy speeding up my speech — should have backed off from 115% to maybe 110%), I can say that it was much improved over the rough cut. And looking back I’m somewhat proud of what I have learned… I had always dreamed of hosting a science/natural history show since I was in middle school! But sometime during high school, I got it into my head that I was too unattractive to even be on camera. Nonetheless, I loved watching PBS (Crockett’s Victory Garden, This Old House, etc.) and I especially loved how the hosts were such approachable people. They weren’t drop-dead gorgeous, but they had a kind of authenticity and gravitas (leavened with humor). I struggled with my camera-phobic block during this class. But then I read an article on the defense lawyer for the Marathon Bomber. The lawyer, Judy Clark, has the kindest face, almost sad. The article talked about how she is very effective at gaining her clients’ trust, and in conveying her clients’ humanity to a jury. That power lies partly in her appearance and demeanor. And I thought I wouldn’t mind having that kind of power — the power to make people trust and understand. I hope that I managed to convey just a little sliver of that in my video (and the one for Science Out Loud as well).
I share this backstory because I find this “beginning” of the video much more interesting than its online beginning. The focus of 20.219 was not on production technique proficiencies, but to have students think about broader learning experiences that the plastic elements of video production could enable, particularly for themselves. There was already visible interaction and learning happening 10 months ago, albeit offline. And it came from the creation of the video itself, before anyone else in the world outside of our tiny, 7-person class knew about what Andrea had made.
The “Official” Beginning
May 21, 2015: “How Do Braces Work” goes live on YouTube. Our birth announcement:
The Middle
(Hopefully you’ve seen the video by this point.)
The End
In the 4.5 months after we released “How Do Braces Work?”, here’s what happened:
May 21, 2015-October 3, 2015, our Tweet receives:
- “Impressions” (the number of unique Twitter accounts to whose feed a Tweet has been delivered, but not necessarily a metric of how many times it was read): 1,083
- Video views (from the Tweet): 1
i.e., The birth announcement was an unimpressive blip in the vast universe of the Internet (it was even worse on Facebook). And aside from shares peaking on the day of publication, nothing too crazy happens — views plug along around 200 or so a day.
Our audience on Day 1 is majority male (70.5%) and 25–34 years old (39%), representative of our overall channel audience (73.6% male, 28% 25–34 years old).
June 17, 2015: Curiosity.com, an edutainment video aggregator and platform (also a Discovery Communications spin-off), Tweets about the episode:
Our daily views bump up to 426 and steadily climb along with other metrics of engagement. In particular, the relative spike in likes and subscribers gained are the most notable upon a quick data glance. We also get a little cluster of increased comments shortly afterward.
This audience is slightly more male than female (53.6%) and somewhat younger (36% 18–24 years and 36% 25–34 years).
August 4, 2015: Over the span of a few days, our daily views jump considerably to 4,924, with the main source of traffic being a video called, “GETTING BRACES ON! ~ Part 1”.
Published on July 8, 2015, it didn’t pick up momentum until early August. My guess is it was lag time for YouTube’s suggested video algorithm to discover it, based on threshold view/share levels. Whatever caused its peak in views, traffic to our video benefitted as a result. Certain metrics go up: views, likes, comments, favorites, videos added to playlists. Shares and subscribers gained don’t jump up with the views. Average percentage viewed noticeably goes down.
This audience has a ton of females (71.6%) and younger viewers (19% 13–17 year, 26% 18–24 years, 23% 25–34 years).
August 25, 2015: Curiosity.com drives a second little peak in traffic, and I’m not sure why. The trend of engagement metrics looks similar to that of June 17th — it appears that Curiosity.com users engage with our video in similar ways.
However, this audience looks a little different from the previous Curiosity.com spike — more females (56.6%) and slightly older (37% 25–34 years, 21% 45–54 years).
September 13, 2015: MythBusters shares the Curiosity.com link that embeds our video (and a couple thousand people share that post). We get 25,539 views overnight.
Audiences directed from the MythBusters Facebook page seem to behave like audiences from Curiosity.com — all the engagement metrics shown in Fig. 3 noticeably spike. And where viewers directed from other YouTube videos on braces tended to watch a lower percentage of our video, viewers from MythBusters watch the most out of any of our viewers.
The demographics of this day’s viewers are very different from the general video viewership (see Fig. 1), but very similar to the TV show’s viewership as of a few years ago. Mostly male (64%), mostly 25–34 year-olds (43%).
September 19, 2015: We get a small boost in traffic again, this time from a YouTube video called “Jana’s First Day with Braces”, which was published on September 9, 2015. Data on daily views for this video is unavailable, but I’m guessing it didn’t take as long as “GETTING BRACES ON! ~ Part 1” to drive traffic to us because SevenPerfectAngels has been an established channel with more subscribers, so it would’ve gotten more views off the bat. The general habits of these viewers seem more similar to those from “GETTING BRACES ON! ~ Part 1” than MythBusters, most notably in their marked decreased percentage viewed but marked increased videos added to playlists.
More females watch our video on this day (61.9%). Viewers are also 29% 25–34 years old.
September 29, 2015: We get another big boost, this time thanks to “Alyssa Gets BRACES”, directing 22,507 viewers to us for a total daily view count of 26,573.
Posted on September 26, 2015 to FamilyFunPack (another established channel with a decent-sized subscriber base of almost 740,000), it experiences a peak in traffic on the 29th (the same day we do). While these viewers click on the video and add it to playlists, they’re not watching very much of it, nor are they really sharing, liking, favoriting, or subscribing.
I found the demographics of our viewership on this day surprising — 52.6% male. A lot of them (41%) 25–34 years. Though their behavior resembles the other viewers who were directed by YouTube video suggestions, their demographics look very different.
The remaining few thousands of views mostly come from “Jana’s First Day with Braces” and other YouTube braces videos. Curiosity.com only accounts for 6 of the views that day.
October 1, 2015: The most overnight views we’ve gotten so far (32,141) with 27,348 coming from “GETTING BRACES ON! ~ Part 1” again. The remaining few thousands of views mostly come from “Jana’s First Day with Braces” and other YouTube braces videos. I have no idea why “GETTING BRACES ON! ~ Part 1” drove traffic again. This day’s audience looks different, though:
- Back on August 4: Ton of females (71.6%) and younger viewers (19% 13–17 year, 26% 18–24 years, 23% 25–34 years)
- This day: Half female (51.2%), mostly 25–34 years (43%)
There are some things this audience has in common with the one from August 4 — they’re not watching very much of the video, but they’re adding it to playlists. They’re not really subscribing or sharing. However, unlike the viewers from August 4, they’re also not liking, commenting, or favoriting. They’re generally less engaged, judging from these metrics.
When I first noticed that the days with the worst audience retention were the days when traffic was driven by other YouTube videos, I admit I assumed it was due to the nature of our audience on those days — teenaged girls with fleeting attention spans. But not only was I wrong about the viewer demographic, the days that have the least amount of engagement — looking at all of the metrics — relative to views (the last big 2 spikes) are the days that have more male viewers and majority 24–35 year olds.
October 3, 2015: The peak drops, but we still get a respectable overnight count of 22,022. Most of the traffic comes from “GETTING BRACES ON! ~ Part 1” with the remaining few thousands of views mostly coming from “Jana’s First Day with Braces” and other YouTube braces videos.
This audience looks and acts like the one from October 1.
Admittedly, this is far from a rigorous analysis of viewer data, especially because in some instances, the sample size is too small to draw meaningful statistical conclusions. What I think is more interesting here are some of the general insights, drawn from the life of “How Do Braces Work?”, that can get us thinking about the success of a video as a conduit for learning:
Single (or a couple) digital engagement metrics on their own provide a limited picture into learning.
Views are obviously not trivial considerations. But for our braces video, the days with the highest view count (last 2 peaks) also had the lowest remaining engagement metrics relative to views. So hopefully the idea that views only provide a very small portion of the engagement landscape it isn’t that hard of a sell.
The other Holy-Grail-ifiable metric is average percentage viewed. MOOC video research in particular seems to be in a retention-centric mode of assessment. But just because someone’s watching doesn’t mean they’re learning, and the reverse is true as well. For instance, it’s true that most of the metrics of engagement (like average percentage viewed) either went down or didn’t peak along with the views on days where other YouTube videos drove traffic to “How Do Braces Work?” But one metric consistently spiked — videos added to playlist. Even though these viewers weren’t watching most of the video, they were adding them to playlists, my guess being the “Watch Later” one in particular. People could have also been adding the video to other playlists that got shared. I don’t currently know how to track how much of those “Water Later” ones actually end up getting watched or how/if these playlists get shared. But regardless of if these instances eventually lead to views or not, it seems like there’s more to examine beyond even views and retention combined.
Offline aspects of how people interact with online video are important, too.
Continuing the notion that one metric is not enough… An initial trend that jumped out at me was the behaviors of Curiosity.com and MythBusters audiences versus ones driven by other YouTube videos on braces. Curiosity and MythBusters audiences generally stayed longer — they watched a higher percentage of the video and notably subscribed to our channel at a relatively higher rate. They also seemed to share at a higher relative rate. I attributed this to them being an already STEM-inclined or interested audience — they got what they came for.
On the other hand, the audience driven from other YouTube videos didn’t watch as much, didn’t subscribe, and especially didn’t share. It may very well be true that they weren’t STEM-inclined, we were facing an uphill battle to change their minds, and just failed to do that with our video. That’s a very relevant and probable challenge. But it could also be less of a reflection of how the video attributes led to a learning experience, but more the viewer’s identity and the way they express that experience. Particularly with shares, what motivates a person to engage doesn’t always have to do with meaningful learning, but the way they craft their online identity. What you share is social currency, and for the STEM-inclined MythBusters audience, sharing our video might’ve earned them some coins in how their peers viewed them.
Maybe the YouTube video audience is learning — they’re still adding the video to playlists and favoriting, after all. But maybe they’re less inclined to share or visibly indicate their interactions; maybe they are learning in a way that our current definitions don’t really accommodate or capture. What’s not to say that once they close the video, they’re not clicking on other videos about human anatomy or Googling something they happened to catch at the beginning? Or has a part of the video made them ask a teacher about the topic the next day? What if they decide to shadow an orthodontist? And what are all the interactions that happen as a result — does that Googling lead to an interest in biology or medicine?
Tracking creator intent to viewer outcomes would be interesting as well — Andrea designed her video to resonate with middle schoolers in particular. Did they make sense of the video (both online and offline) like she thought or hoped they would? And how do those interactions differ from those of the crucial audience that made this take off (The MythBusters and their followers)?
Additionally, a YouTube user profile doesn’t always reflect the actual viewer — it could be a child using a parent’s iPad, for instance. This might be less of an issue with MOOCs, where there’s more consistency between the registered and actual user. Nonetheless, in-depth interviews might prove useful as another way of teasing out meaningful information where YouTube demographic data falls short.
Outcomes are a collection of measures — not knowledge retention, viewership, comments, shares, etc. alone (or in limited combinations). A learner’s identity and how it affects the way they make sense of an experience with media reveals itself in a constellation of metrics, many offline.
Numbers sometimes lie. (Or at least only tell a tiny part of the story.)
Along the lines of “just because someone’s watching doesn’t mean they’re learning, and the reverse is true as well,” I’d also argue that just because someone’s commenting doesn’t mean they’re learning, and the reverse is also true. The scale and variation is very small in the comment numbers for our video. So instead of relying on numbers, which could lead you to think that the August 4 audience is learning a lot (more commenting activity despite a drop in viewer retention), I looked at the actual comments themselves.
They are overwhelmingly consistent, across age and gender on YouTube and on the MythBusters’ Facebook post. They’re some version of either “So that’s how they worked!” (sense of wonder) or “Yeah when I had braces, XYZ happened” (anecdotal). Very few (2 of 44 on YouTube) are inquiry-based. None (that I saw, at least) are about bone remodeling’s applications to bioengineering research bit at the end.
Is this just a topic that doesn’t lend itself to much investigation afterward? Perhaps it does, but in a way that people don’t feel inclined to make visible through their comments. Was the way we produced the video not optimized for discussion? I’ve often wondered that about our videos in general. One approach could be having our hosts more directly address the audience and ask them questions and observing how that affects comment types, in the context of many other metrics.
Comment types can also be a product of the type of video as well — certain videos are more “school-friendly” (tackling a topic that’s hard to teach) and designed for more seamless integration into a classroom curriculum.
For instance, our video on angular momentum (a completely different series from Science Out Loud) doesn’t have many comments, but the ones that are there are probing and clarification questions like “What if you’re not standing on a wheel? How is angular momentum conserved then?” (another user jumped in to answer).
Other videos tackle concepts that are hard to relate and explain outside of school, too. Our video, “How Computers Compute” received some comments along the lines of, “Finally something I can share to my non-techie friends to help them understand what I do/love,” which isn’t a type of comment we see too often.
And finally, “Fire + Sand = Glass!” is one of the very first videos that came out of our program, back when students produced videos entirely on their own. It’s the one video on our channel whose view count surpasses Andrea’s (though it’s been on YouTube 3 years longer) and gets what I’d call “curiosity questions.” These are ones that take the presented material and try to extrapolate further, simply out of curiosity (and not because they’d need to know it for a test for any reason). “Why is the thin end needed?” “What about molten glass dropped in liquid nitrogen?” “Can you make a Prince Rupert’s drop out of metal?” They’re the most overtly learning-oriented comment types.
Looking at the comments themselves might give us more insight into what viewers are getting out of watching a video, and how we might go about creating videos to help people make sense of them in a certain way. But just because something elicit more comments or a certain type of comment doesn’t mean it’s evidence of superior learning. I think learning is happening with all of these videos, but that the type of learning is different. There’s a lot left to explore in comparing the types of comments to other attributes of the video, the platform itself, and other metrics to better understand these types of learning.
In addition to expanding our notion of “outcomes,” we also should consider outcomes as a result of the entire lifespan of a video, not just its middle.
I also hope we also expand our notion of “The Beginning” as a space for meaningful learning experiences for media creators, based on student experiences in 20.219. How do people learn and interact during the creation of media, how does that impact the final product, and how does that impact the way other people learn and interact with the video? In particular, targeting videos specifically to the non STEM-inclined might benefit from this holistic look at a video’s life. Andrea struggled at the beginning of class, and as a result, let what personally resonated with her (and, she hoped, a target audience) guide her topic. Designed to hook audiences using an appeal to personal experiences and interests, we might call the video a “success” looking at the audiences driven by suggested YouTube videos. And the way we refocused her ending on the unexpected element— that is, braces work by causing bone remodeling — seemed to strike a chord with Curiosity.com users and MythBusters followers. Would audiences have interacted differently with a video more akin to Andrea’s very first idea iteration, one that was more information and terminology-driven? That may be something to look into. We sure would’ve missed out on the Tweet that shared the Curiosity page along with, “What the f***. MY MOUTH BROKE ITSELF D:” though.
And as for the “end”, it appears that some of these engagements are less a product of the video itself as it is a result of the platform, the users the platform attracts, and/or the community in which people can interact with the video. All the viewer data for this video came from our YouTube analytics account — I wouldn’t be surprised to see all of these trends go out the window on iTunesU, Khan Academy, PBS Learning Media, etc. which tend to self-select for a STEM-inclined user and have different modes of video interaction. My next step would be to look at how users on these platforms are engaging with this video.
For instance, the vast majority of the audience directed from MythBusters watched the video on the Curiosity.com landing page, which features “Key Facts to Know” and suggested videos that an editorial team assembles. Key Facts, or shareable ones, relate elements that we very intentionally built into the video. It’s a quality that might be a tenant of informal, YouTube-based science videos and a requirement that separates them from lecture-style or MOOC videos. But how important is it, actually, to have that shareable fact element and cause people to reexamine what they thought they knew about something? A shareable fact, at its core, is also an unexpected one. Does that tangibly add to a different kind of learning experience? Can those additions inform how we create videos to better facilitate learning? Can that relate to why online multimedia that includes a discussion of misconceptions seem to lead to better learning outcomes?
By being on the web, a video is thrown into the realm of new media — it becomes a digital object not simply to consume, but with which one can interact. And the extent to which we as content creators and/or consumers actually take advantage of this designation (i.e., why and how we interact with digital material, if we even interact at all) can vastly change the life and effective lifespan of video. What seems evident is that these 4 minutes are just a tiny snapshot into the interactions and learning related to this video, a small fragment of its actual life. The middle is the smallest chapter in the story of this video. I understand that some of the investigations I’ve suggested aren’t always practical, or even useful. At the end of the day, content creators have to decide for themselves how they want to define learning. But hopefully the life of this video at least gets us expanding our notions of learning to more fully embrace video as new media.
