Yet another implementation of GLTextureView

Using Jetpack Compose and Coroutines. Wow.

Introduction

Looking at Android’s documentation, TextureView seems like something beautiful to anybody who wants to have the advantages of a GLSurfaceView but with extra features like transparency handling. That was exactly our case in this article since we wanted to render an OpenGL output in a view that could handle transparency.

Hopefully, there are several implementations over the internet but it’s quite hard to find either documentation on how it is achieved, what is done or how the implementation ended up like this. Usually, the code from GLSurfaceView is copy/pasted directly in a class inheriting from TextureView and all the fingers are crossed for it not to break in production for unknown reasons.

This seemed not acceptable since the documentation lets us think that the implementation should be seamless. And it is when we dig it a bit!

Digging information

TextureView

According to the documentation, all we have to do is to draw on a SurfaceTexture provided by the TextureView. The TextureView should then handle the rest on its own. The SurfaceTexture is provided by the TextureView to a TextureView.SurfaceTextureListener so we should not need so much wiring on this side.

OpenGL

This part is trickier since we will have to reverse engineer the implementation of GLSurfaceView to understand how it can draw inside a SurfaceTexture. There are 2 major concerns about wiring OpenGL on Android :

• EGL (a kind of “bridge” between the OS and OpenGL) handles wiring between OpenGL and the SurfaceTexture
• All the EGL surface handling commands and OpenGL drawing commands MUST happen on the same thread.

For the first point, we mainly inspired ourselves from the work of a developper who wrote a 3-parts article (here translated from chinese : 1, 2, 3) and the work of another dev who came out with an existing implementation. It seems that EGL is the one which takes SurfaceTexture as an input and the rest is just a bunch of methods for configuration and actions that have to be done in a specific order. We should then just have to keep this specific order and reduce/simplify the amount of configuration needed to get a noob-friendly implementation of our TextureView.

The latest point is important since while we want to use coroutines to get rid of the notion of threads, we must keep in mind that we will have to restrict our CoroutineContext to a single thread.

Implementation

Architecture

We want the architecture to clearly separate responsibilities between the OpenGL and EGL libs and TextureView as well. We also want the API to the outside world to be very simple.

Fig.1 We want this API

The API is a Composable function that accepts a Modifier like any other Composable, a Renderer that will implement the drawing part and a requestRender Flow that would trigger a new rendering each time the outside would emit a Unit in it.

To separate the concerns, this composable will have an inner GLTextureViewModel that will handle the wiring between the TextureView and the other components.

Fig.2 Ideal architecture of our GLTexture

Implementation

GLTexture Composable

Fig.3 Actual GLTexture implementation

The implementation of the Composable is really simple here. It just defines an AndroidView in a Box that hosts the TextureView. Its surfaceTextureListener is defined to be the viewModel of and a CoroutineScope is passed as well as an onDispose call when the Composable is being disposed of. The renderer is also passed to the viewModel so that it can wire it as well. This way, the viewModel should have everything it needs to :

• Have a SurfaceTexture to pass to EGL thanks to being a surfaceTextureListener
• Have a parent CoroutineScope to create an “OpenGL scope” from it
• Know when to dispose of OpenGL and EGL resources
• Render drawings using the renderer that will interact with OpenGL

Wiring in GLTextureViewModel

Fig.4 The ViewModel’s structure

As expected, we find here the implementations of the TextureView.SurfaceTextureListener’s 4 methods, an onDispose method and a drawFrame method that is run each time an event is received in the requestRender Flow.

We can also notice 2 attributes in this class :

• openGLContext : an instance of a class that hosts a reference to the single thread OpenGL and EGL require.
• currentEGLHandler : a handler to interact with EGL with helper methods. It will be created as soon as the viewmodel receives a SurfaceTexture in onSurfaceTextureAvailable.

We will now go through each part of this view model.

OpenGLContext

Fig.5 Actual OpenGLContext implementation

It is a simple wrapper around a MutableSharedFlow whose events are listened to in a CoroutineScope whose CoroutineContext has a single thread and which inherits from the Composable’s CoroutineScope.

TextureView.SurfaceTextureListener

Fig.6 TextureView.SurfaceTextureListener interface implementation

This implementation is quite straightforward :

onSurfaceTextureAvailable :

• create a EGLHandler with the SurfaceTexture
• notify the renderer of the creation of the surface
• notify the renderer of the size of the created surface
• ask the renderer to draw the first frame

onSurfaceTextureSizeChanged :

• notify the renderer of the new size

onSurfaceTextureDestroyed :

• do nothing since every cleanup will be done on disposing of the composable

onSurfaceTextureUpdated :

• do nothing. Maybe we should update the surface in the EGLHandler. So far we have had no problem with this.

We can notice that every call to the EGLHandler or to the renderer must be done inside the OpenGLContext.

Drawing frames

Fig.7 Subscription to the requestRender events

Fig.8 Frame drawing logic

During the initialization of the view model, it subscribes directly to the requestRender events and routes it to the drawing method. Drawing is quite straightforward as well : requesting the renderer to draw and tell the EGL handler to display what OpenGL just drew.

Disposing of our garbage

Fig.9 Cleanup logic

Eventually we want to tell the renderer and the EGL handler to do its own cleanup as well. At this time, we then get rid of the EGL handler reference to prevent any further calls to it after this cleanup step.

Handling EGL

Fig.10 EGLHandler’s structure

We want the handler to expose 2 attributes and 2 methods :

• gl : OpenGL context for usage in the renderer.
• config : The config used by EGL for the current context. It is intended to be used by the renderer as well.
• destroy() : called to get rid of the resources of EGL.
• displaySurface() : called to display what is currently drawn by OpenGL on the surface provided in the constructor of the EGLHandler.

To implement everything, we need to isolate and rearrange the instruction gathered in the implementation of GLSurfaceView to make it more readable and separated by concerns. This is a work of tracking down the responsibility of every line in GLSurfaceView and pasting it here in the proper place.

We also put all the calls to EGL that are intended to initialize every attribute needed by this class in a EGLHandlerBuilder. It is also a gathering and re-arrangement of instructions found in GLSurfaceView.

Fig.11 The EGLHandlerBuilder’s structure

Usage

This GLTexture can be used to display any OpenGL drawing in a composable view. In our project, we use it to display videos that handle transparency using OpenGL shaders. We explained how we did it in this article.

Fig.12 Usage of GLTexture to display videos edited using OpenGL

Fig.13 Example of a GLTexture showing an transparent video on a map

Real world example

In this repository we made an implementation of the GLTexture Composable within an library that can display transparent videos.