Netflix’s “Skip Intro” Feature — How the Hell do They do That?
If you are a Netflix fan (as I am sure most are) you may have noticed in the past 6 months or so that you can skip intros of most shows. Usually shows that have cold openings.
I always wondered how they do this, because it’s a huge time saver and makes situations like binge watching especially very fluid.
Here are some of my guesses on how they do this.
People, People, People
Some poor sucker(s) go through thousands of hours of content and manually timestamp the beginning and end of intros. Netflix, I really hope you don’t do your peeps like that.
Audio Tagging
This is the most likely method. Netflix takes the audio sample of a show’s intro — Friends for example, and queries all the episodes to find where that audio sample appears. For accuracy I’m sure they can apply conditions like only looking for the audio in the first 2 mins. To be even more accurate to make sure the intro’s beginning has the shows’ title for example, apply a little computer vision.
Machine Learning
The company is all about machine learning, from understanding what users like, don’t like, tracking behavior, offering recommendations (where I think Netflix sucks btw, on that on a later post) etc. So I won’t be surprised if they apply it for the “skip intro” feature as well.
I can envision for example Netflix learning just from the show Friends (or a base show) and teaching a machine to identify what an intro is without feeding it the audio sample or any visual cues.
So the machine learning algo will discover on its own to apply what it learned from Friends (a base show) to a show like Fraiser or The Office. It will identify an audio sample appearing on all episodes (pattern) around the same time (condition) mixed with some visual cues (condition) like the show’s title.
User Signal & Feedback
It’s possible Netflix just takes user signals to do this. Like by tracking where users are manually skipping and taking that data to base the feature. But I highly doubt this. Seems too much work and I really don’t know how accurate it would be.
I do know however that Netflix takes user feedback to improve the accuracy of this feature. By asking users to let them know if the “intro skip” feature is not working properly.
