Caption Overview

Taku Semba
Mar 10 · 3 min read

Captions provide viewers with information about what is occurring in their videos for viewers who are not able to understand or hear the language, and many streaming services, such as Netflix, YouTube, and Hulu, support captions to increase accessibility.

Open Captions vs Closed Captions

There are mainly two types of ways to support captions. One is Open Captions and the other one is Closed Captions.

Open Captions

Open captions, also known as burned-in, baked, burnt or hard-coded captions, are expressed as a part of a video frame, which means it can not be turned on and off. While it does not provide any flexibility, it is very easy to use because you do not need to keep and maintain additional caption files.

Closed Captions

On the other hand, since closed captions treat the caption data separately, it provides the flexibility of switching captions on/off and selecting captions for a specific language. Because of this flexibility, closed captions are widely used over open captions. However, unlike open captions, there are multiple ways to support closed captions and a media player has to support it to display the captions.

Options for Closed Captions

As I described above, there are many ways to carry caption data and a media player has to support some caption formats to archive closed captions. In this post, I will use ExoPlayer and deal with main caption formats supported by ExoPlayer.

You can confirm what type of caption formats are supported by ExoPlayer below.

TTML

TTML (Timed Text Markup Language) is an XML based caption developed by the World Wide Web Consortium (W3C) and supports so many features such as positioning, alignment, and styling.

Once you have the TTML file, you can show the captions using ExoPlayer.

If you are interested in how and where ExoPlayer parses a TTML file, you can check TtmlDecoder.java.

WebVTT

WebVTT (The Web Video Text Tracks Format) is also developed by W3C, but it’s simpler compared to TTML format.

Once you have the WebVTT file, you can show the captions using ExoPlayer.

If you are interested in how and where ExoPlayer parses a WebVTT file, you can check WebvttDecoder.java.

If you use mp4 file-based streaming, you can put caption data into a box. How you can put those caption data into a box can be confirmed from ISO/IEC 14496–30.

CEA-608/CEA-708

CEA-608 was developed first for broadcast television, but CEA-708 was introduced to support more languages, appearances, positionings, etc. To use CEA-608/708 in ExoPlayer, caption data have to be embedded in SEI messages in fmp4 video streams, which means, instead of getting a caption file separately, caption data can be retrieved from SEI in NAL Unit.

You can play a media file with CEA-608 embedded from this link.

Summary

In this post, I explained the differences between Open Captions and Closed Captions and looked at some of the formats for Closed Captions, which are TTML, WebVTT, CEA-608/CEA-708. I hope this will help someone understand the overview of how captions work.

Taku Semba

Written by

Android Engineer at AbemaTV Github: https://github.com/takusemba

More From Medium

More from Taku Semba

More from Taku Semba

Deep Understanding of Seek

68

Related reads

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade