External texture rendering with Unity and Android

One of the difficulty faced in plug-in development in Unity for android is the ability to update or modify the content of Unity texture from Android plug-in code. Not many people got it working the right way. It would break if the Unity version changes or if the Android version changes. Some of the approaches which does work, are kind of hacks and they are bound to break. Some of them are resource intensive as they tend to draw or copy the content from render buffer which is ineffective.

In this article we are going to see a fool proof and simple optimised mechanism to update the texture created by Unity from android.

The key to the solution is to understand how OpenGL and Unity Rendering works. There are 2 key elements

1. The OpenGL Context
2. The Unity Rendering Thread

We all know that Unity uses multithreading to support coroutines. The rendering itself can be “Single” or “Multithreaded rendering”. First, we will see the Single threaded rendering model and later on extend the solution for multithreaded rendering. Before we proceed to the solution, please make sure that the “Multithreaded Rendering” is disabled in Player settings

Creating Unity texture is simple. You can get the underlying native OpenGL texture pointer using “GetNativeTexturePtr” method like below

_imageTexture2D = new Texture2D(1280, 800, TextureFormat.ARGB32, false)
_nativeTexturePointer = _imageTexture2D.GetNativeTexturePtr();

In order to successfully create an Android SurfaceTexture from Unity, all that you need is the texture pointer which can be obtained as mentioned above. But the SurfaceTexture created like that doesn’t work. The reason being, the creation of SurfaceTexture need to happen in the render thread and when the correct OpenGL context is active, which is not the case always. Most of the plug-in calls are made from coroutine threads created by Unity and doesn’t have the correct OpenGL context and hence they fail.

So, the right approach is to initialise the SurfaceTexture when the correct OpenGL context is active. Here’s the correct plug-in snippet

private void initSurface() {
    unityContext = EGL14.eglGetCurrentContext();
    unityDisplay = EGL14.eglGetCurrentDisplay();
    unityDrawSurface = EGL14.eglGetCurrentSurface(EGL14.EGL_DRAW);
    unityReadSurface = EGL14.eglGetCurrentSurface(EGL14.EGL_READ);

    if (unityContext == EGL14.EGL_NO_CONTEXT) {
        Log.e(TAG, "UnityEGLContext is invalid -> Most probably wrong thread");
    }

    EGL14.eglMakeCurrent(unityDisplay, unityDrawSurface, unityReadSurface, unityContext);

    GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
    GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, unityTextureID);
    GLES20.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_LINEAR);
    GLES20.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR);
    GLES20.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_WRAP_S, GLES20.GL_CLAMP_TO_EDGE);
    GLES20.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_WRAP_T, GLES20.GL_CLAMP_TO_EDGE);
    mSurfaceTexture = new SurfaceTexture(unityTextureID);
    mSurfaceTexture.setDefaultBufferSize(mTextureWidth, mTextureHeight);
    mSurface = new Surface(mSurfaceTexture);
    mSurfaceTexture.setOnFrameAvailableListener(this);
}

The “initSurface” plug-in call must be made from a render thread from Unity. In a “Single” threaded rendering, the “Update” method of MonoBehaviour is always executed in “UnityMain” which is a render thread. So it makes sense to call the initialisation in Update method.

private void Update()
{
    if (_androidApiInstance == null)
    {
        // it is important to call this in update method. Single Threaded Rendering will run in UnityMain Thread
        InitializeAndroidSurface(1280, 800);
    }
    else
    {
        _androidApiInstance.Call("updateSurfaceTexture");
    }
}

public void InitializeAndroidSurface( int viewportWidth, int viewportHeight)
{

    AndroidJavaClass androidWebViewApiClass =
        new AndroidJavaClass("com.thoughtworks.texturerendererandroidplugin.TextureRendererPlugIn");
    
    AndroidJavaClass playerClass = new AndroidJavaClass("com.unity3d.player.UnityPlayer");

    AndroidJavaObject currentActivityObject = playerClass.GetStatic<AndroidJavaObject>("currentActivity");

    _androidApiInstance =
        androidWebViewApiClass.CallStatic<AndroidJavaObject>("Instance", currentActivityObject,
            viewportWidth, viewportHeight,_nativeTexturePointer.ToInt32());
}

Note: If you try to move the “InitializeAndroidSurface” from Update method to “Start” method, the solution will fail since the Start method doesn’t run in a render thread.

The complete Android Plug-in class looks like below

This plugin tries to create a SurfaceTexture with Unity’s textureid and draws random circles in the texture every 100 milliseconds.

The next key thing is that the “updateTexImage” of SurfaceTexture need to be called only from Render thread. When called from any other thread, the texture updation will fail. So we make sure that the “updateSurfaceTexture” method from the plugin also gets called from the render thread (“Update” method). Complete Unity Code snippet

The RawImage for the script need to be set from the scene. And this RawImage would reflect all the updates to the SurfaceTexture. This is how it looked like in the android emulator.

Now the solution discussed so far works only for the “Single” threaded rendering model in Unity. To support “Multithreaded” rendering, we need to employ JNI and use “GL.IssuePluginEvent”. In a multi threaded rendering context, the IssuePluginEvent method makes sure that the method supplied to it is always called from Unity’s current render thread. So, in JNI we need to attach the correct java plugin method to this render thread and execute it. You may refer the sample provided by Unity here.

The solution outlined here also works good in a multi threaded context provided we write the appropriate JNI methods and call them via GL.IssuePluginEvent. Since that involves bit more understanding of JNI, deferring that for another article if more people request for it.

Thanks for reading. Appreciate your comments.