HLS playback in ExoPlayer

ExoPlayer’s ultimate goal is to create compelling and stable playback experiences. These experiences can be thought as a combined effort between the player itself and the content provider. The purpose of this post is to shed some light on what ExoPlayer considers to be high quality HLS streams. In addition, some common error conditions are presented along with what behaviors to expect and, to conclude, we briefly list planned future work.

General good practices

Use precise segment durations

Version 6 of the HLS specification introduced floating point durations for segments in media playlists. Before then, durations had to be rounded to the nearest second. Incorrect segment durations make it difficult to seek a point in playback, and anything that makes seeking difficult also affects switching between variants. Enough accumulated error can even trick the player to stop loading data, triggering endless buffering.

Avoid structural changes in segments

ExoPlayer does not currently handle changes in the number of available tracks. As an example, the removal of the audio track from the stream will most likely cause a fatal failure. A workaround for this is adding a silent audio track or a placeholder empty video in the gaps.

Use independent segments

For segments that contain video, the specification states:

A segment that contains video SHOULD have at least one key frame and enough information to completely initialize a video decoder.

There are no further constraints regarding cross-segment dependencies. It is not uncommon to come across HLS streams with sub-optimal key frame positioning, which causes undesired behaviors when seeking and extra segment downloads when switching variants. For example:

• If a seek happens to position t and segment s (that contains the video frame for t) is downloaded, it is possible that the first key frame in s appears after t, meaning that the video frame at t cannot be decoded. The observed behavior will be a frozen frame for a short period of time, until audio catches up with the frame, after which playback will continue normally.
• In the case of a variant switch, ExoPlayer prevents playback discontinuities by downloading the segment preceding the one that contains the video frame at which the switch occurs. Naturally, this means discarding all frames up to the target frame.

Fortunately, this is all avoidable. First of all, by segmenting content so that the first frame of every segment is a key frame, the first scenario is avoided. And second, version 13 introduced #EXT-X-INDEPENDENT-SEGMENTS, which lets the player know (without downloading any segments) that the content is segmented in this way. With this information, ExoPlayer can save one segment download per variant switch.

Avoid multiplexed streams

ExoPlayer does not support multiple streams of the same type muxed into a single Transport Stream (note this only applies to HLS). On top of this, as a general rule, downloading any streams that are not played is a waste of bandwidth (particularly true for audio and video streams). The corollary is: Content providers using ExoPlayer should try to serve every track demuxed. Not only will this potentially save bandwidth, but will also allow ExoPlayer to provide support for every available language that the playlist serves.

Include complete (and correct) information in the master playlist

Master playlists can provide a good deal of information that the player can use during preparation and track selection.

#EXT-X-STREAM-INF attributes:

• Average bandwidth,
• codecs,
• resolution,
• frame rate,
• and Closed-captions (indicating “NONE” if there are none in the stream).

#EXT-X-MEDIA attributes:

• Language,
• selection flags (DEFAULT,AUTOSELECT,FORCED),
• and number of channels.

Including these attributes allows ExoPlayer to safely (and efficiently, as it could save a bunch of segment downloads) decide whether the device is capable of decoding the content and, on top of this, to perform good track selections. For example by selecting a video track with a resolution appropriate for the device's screen. Any downloads made before the initial track selection might have to be discarded, so it's convenient to get it right the first time. Depending on the stream, this might result in faster playback startup times.

Good practices in live streams

The HLS specification defines a set of rules for modifying media playlists. The following list of recommended practices applies to streams that can be modified according to any of those rules. That is, segments are both added and removed from the playlists.

Include #EXT-X-PROGRAM-DATE-TIME

Having this tag in every snapshot of the media playlists along with precise segment durations makes it possible for the player to know exactly which period of time each segment belongs to, allowing it to map any playback position to the correct segment. This can help the player to avoid unnecessary segment downloads and playback discontinuities.

Include the discontinuity sequence number

Version 2 of the HLS spec introduced the #EXT-X-DISCONTINUITY tag. It’s used to indicate changes in the file format, timestamp sequence and other characteristics. It is fairly common in streams that include ads. In version 12 of the spec, the #EXT-X-DISCONTINUITY-SEQUENCE was introduced. This tag provides the only way to safely and seamlessly switch to a different variant in a live stream containing discontinuities. If it’s omitted, ExoPlayer falls back to a best effort approach by keeping track of contiguous segments when possible, and using the last updated playlist as reference when the first alternative is not possible. An incorrect guess can cause position jumps in playback, sample discontinuities and, worst of all, the dreaded endless buffering state. To make matters much worse, it is extremely hard to reproduce under regular circumstances. Good news: It shouldn’t be hard to add support for #EXT-X-DISCONTINUITY-SEQUENCE to your content.

Use a long live window

In live streams, it is risky for playback to be close to either end of the window. Being too close to the live edge might mean the player has to wait for a new segment to be appended to the playlist, which the user will perceive as a rebuffer. Being too close to the start of the window might cause the loading to fall behind, which will require a repreparation of the player (look for BehindLiveWindowException in the demo app's PlayerActivity). Short live windows have two ill effects:

1. The playback position is never far from either edge.
2. They require much more frequent playlist refreshes.

In addition to these points, long live windows allow the user to comfortably seek in the stream. We recommend a minimum length of one minute for live windows.

Common error conditions

This section explains how certain error situations are handled by ExoPlayer.

404/410 HTTP response codes

When the server returns a 404 while trying to load a Media Playlist or a segment, ExoPlayer tries to continue playback seamlessly by downloading alternative variants. The error will be propagated if the player cannot proceed with downloading any variant.

5xx and other HTTP response codes

Loading errors other than HTTP 404 and 410 are not considered clear indication that a playlist has been removed, and so the player does not attempt to switch to another variant. These error codes are propagated as soon as playback cannot continue. In response to developers' requests, this behavior will be customizable by the app in the future.

Media sequence step back & Playlist reset

According to the spec, the #EXT-X-MEDIA-SEQUENCE is not allowed to decrease. However, there are certain scenarios in which this might happen. Server load balancing is an example of this. From time to time, streams might be restarted as well.

Media sequence decreases with segment overlapping are handled seamlessly by ExoPlayer, under the assumption of just being an older version of the same playlist. On the other hand, if there is no segment overlapping, ExoPlayer assumes the stream has been reset and propagates a PlaylistResetException. It is up to the app to reset the player to continue playback.

Stuck Playlist

As a consequence of reports from content distributors, ExoPlayer is on the watch for stuck playlists, caused by server side errors. A live stream playlist is considered to be stuck if it is not updated in 3.5 times its segment target duration. If the player meets this condition, it will try switching to other equivalent playlist and will eventually propagate a PlaylistStuckException if it is not possible to proceed. Note that this error detection mechanism benefits from a long live window.

Playlist parsing error

This is a fatal error. The player will propagate the error immediately if it occurs during preparation, or as soon as it enters the buffering state.

HLS work in progress

As the last section of this post, an incomplete but brief list of the planned future work:

CMAF/CENC support (#1661)

CMAF support for HLS is missing some final touches related to CENC DRM.

Extractor injection (#2748)

It is currently not possible to use custom extractors in the HLS module without modifying the library code. Extractor injection will allow app developers to cleanly customize the way in which segments are extracted. A common use case is to change the TsExtractor’s flags and modes.

Playlist-only preparation (#3149)

The objective is to prevent segment downloads while preparing the player. Once implemented, playlist-only preparation will retrieve all track information from the master playlist, saving the segments' bandwidth and, by doing so, potentially reducing the initial buffering duration.

Multiple video groups support (#2600)

This has been on ExoPlayer's road map for some time. This will allow delivering multiple video track groups. Each of them could present different camera angles or, for example, different parts of the pitch in a sports event.

Load error behavior customization (#2844)

Last but not least, a feature that will affect not only HLS but potentially other MediaSources as well. By customizing load error behavior, the developer can choose to avoid unnecessary retries (example: authorization errors or FileNotFoundExceptions), or even blacklist specific error codes to allow seamless playback continuation (for example, by blacklisting 5xx HTTP error codes when different variants are served by different servers).

Ideally, this post has provided some practical pointers to improve apps delivering HLS streams using ExoPlayer. Let us know in the comments about your experiences and what customization opportunities you are looking forward to!