ExoPlayer for building powerful VR players on Cardboard and GearVR
A year ago we started to build our own VR app with the aim of creating a social movie theater where you can watch videos with friends across the world. One of the key features is the player itself, so we decided to prototype a Cardboard app based on the Android MediaPlayer first and then ExoPlayer. This was the foundation of our Unity VR Plugin in order to have a solid and reliable player which supports HLS, DASH or SmoothStreaming adaptive playbacks (not currently supported by the MediaPlayer API) with or without DRM support.
Prototyping with Cardboard and ExoPlayer v1.5.x
We first prototyped a simple app using the native Cardboard API for Android with ExoPlayer for playing local and streaming videos on an OpenGL texture. We created a basic 3D scene using OpenGL ES for displaying a video and control the playback. Based on ExoPlayer demo app, we can easily create a wrapper which takes all the data needed to stream a content such as the video and subtitles URLs or the video metadata. The SurfaceTexture is extracted from the OpenGL texture created to render the screen and passed to the ExoPlayer wrapper in order to display the video. And that’s it, we now have a simple player where you can stream a large set of video type.
Below you can see a screenshot from our sample app with basic controllers play/pause and fastforward/rewind.
ExoPlayer plugin for Unity
As we wanted to add a more complex 3D scene in order to have a realistic and more immersive experience, we decided to build our project using Unity for GearVR at first (Google VR, Oculus and VIVE will be available shortly). We rapidly faced the limitations of the Unity MovieTexture to play videos on Android, so we decided to port the ExoPlayer wrapper used for the Cardboard app on Unity.
We built an Android library, based on 3 external libraries (ExoPlayer, OkHttp3 and Okio), exported as a .aar file, to handle the player features in Unity. You can communicate with the Android API or an Android library by using the Unity bridge. The following example describes how to get the current activity, initialize an AndroidJavaObject and call methods from this object outside or within the Android UI thread. If you want to know more about building plugins for Android, have a look here.
Rendering a video in our Unity scene is possible thanks to some work with JNI which allows us to get the surface pointer from the screen texture and then pass it to the plugin in order to render the video. The following example is a basic use of the JNI bridge for calling methods on the Android side:
Our plugin allows us to play adaptive streaming or basic video format, add side-loaded subtitles (VTT or SRT), manage playlists, watch 360° videos or add a DRM support for protected content. The player states and info are sent through the plugin listeners in order to manage the video playback and UI updates on the Unity side. All of this thanks to the ExoPlayer library.
In conclusion, we managed to build a powerful video plugin based on ExoPlayer v1.5.x (now supporting ExoPlayer v2.x) for Unity allowing us to develop VR video apps for Cardboard and GearVR. We shall post more details about our VR projects in the next few weeks. Stay tuned!
Visit https://cinemur.pro/
