Tunneled video playback in ExoPlayer
From Lollipop (API level 21), Android device manufacturers have been able to implement an optional feature called tunneled video playback. Primarily intended for Android TVs, tunneled video playback promises benefits such as better audio/video synchronization (AV sync) and smoother playback.
In ExoPlayer 2.2 we’ve added support for enabling tunneled video playback on devices that support it. This post explains what tunneled video playback is, describes its benefits and drawbacks, and shows how it can be enabled when playing video with ExoPlayer.
What is tunneled video playback?
To understand what tunneled video playback is, it’s first helpful to understand how ExoPlayer interacts with Android’s low level media APIs during regular (i.e. non-tunneled) video playback. The diagram below shows how audio and video samples flow between ExoPlayer components (shown in blue) and Android platform components (in gray).
ExoPlayer obtains encoded audio and video samples from the MediaSource being played. These samples are queued to decoders exposed by Android via the MediaCodec API, and decoded samples are dequeued. For audio the decoded samples are then written to Android’s AudioTrack API for output. For video, MediaCodec.releaseOutputBuffer is called to render each video frame onto a Surface. ExoPlayer is responsible for AV sync, which is achieved by polling the AudioTrack for its current position and timing the release of each video frame to occur at the correct time.
In contrast, the diagram below shows how ExoPlayer interacts with Android’s low level media APIs when tunneled video playback is enabled.
In tunneled mode ExoPlayer does not dequeue samples from the video decoder. After the encoded video samples are queued, decoding and rendering is handled entirely by Android. The underlying platform also assumes responsibility for AV sync. ExoPlayer adds special sync headers, which contain presentation timestamps, to the data written to AudioTrack. The platform uses these timestamps together with timestamps passed when queuing the encoded video samples to ensure that audio and video are played out in sync.
What are the benefits of tunneled video playback?
Tunneled video playback makes it possible for device manufacturers to leverage hardware video decoding pipelines for playback. Such pipelines are common in system on chip (SOC) solutions designed for TVs. For this reason tunneled video playback is most often supported on Android TV devices. Leveraging a hardware video decoding pipeline can bring a number of benefits compared to what’s possible for non-tunneled playback, including:
- Better AV sync, both in terms of accuracy and precision.
- Smoother video playback. This may include hardware based frame rate conversion if supported by the pipeline.
- Lower CPU load since work is offloaded to the pipeline.
- Better support for 4K and high frame rate content.
- Support for HDR, as documented here.
What are the drawbacks of tunneled video playback?
It’s important to note that tunneled video playback isn’t suitable for all use cases. It’s significantly less flexible than non-tunneled playback. In particular, tunneled video playback:
- Can only output to a SurfaceView, and is therefore unsuitable for use cases where SurfaceView does not provide sufficient flexibility, such as VR and 360⁰ video playbacks.
- Can only be used when the media being played includes both audio and video.
- Typically supports only a subset of the video codecs supported by non-tunneled video playback.
Furthermore, it is generally not possible to transition seamlessly between tunneled and non-tunneled video playback. The output Surface may flash black when such a transition occurs. Hence it’s preferable to avoid enabling tunneling if a switch to non-tunneled playback may be required at some point during the playback session.
Enabling tunneled video playback in ExoPlayer
If tunneled video playback is suitable for your use case, you can enable it in ExoPlayer with the following call on DefaultTrackSelector:
You should only enable tunneling if targeting a SurfaceView, however there is no need to explicitly check whether the current media includes both audio and video or whether the device supports tunneling for the relevant video codec. ExoPlayer will only enable tunneling if the above call has been made and if the necessary conditions are met. Furthermore tunneling will be automatically disabled and re-enabled as necessary should conditions change.
We hope supporting tunneled video playback will enable higher quality playback experiences on Android TV. Give it a try and let us know how you get on in the comments section below. If you encounter an issue, please report it on our Github issue tracker. Good luck!
