Playing Video in ARCore using OpengGL

Update 20/August/2018 : Support for playing video in sceneform added in sceneform1.4, see here

For playing videos using sceneform sdk see this sample

Screenshot

Google released Sceneform but it doesn’t support VideoView to play video even though it gave support for Android Views with ViewRenderable ,we have to use OpenGL.

You can try Sceneform from here : https://github.com/google-ar/sceneform-android-sdk.

We are going to modify the ARCore samples project : hello_ar_java from ARCore SDK repo, you can download it from here : https://github.com/google-ar/arcore-android-sdk.

As you can see in project structure Render classes are the classes which are placing object in AR Scene.

We are going to create a new ViewRenderer.java which renders video in AR Scence.

ViewRenderer.java contains 4 main methods : createOnGLThread(),update(), draw(),start().

Initialize every thing in createOnGLThread method().

by calling update() method you can change model matrix, you can scale and translate video by updating model matrix here.

draw() renders video by getting view matrix and projection matrix from ARCore Camera (frame.getCamera()).

we update texture and set dimensions for video frame, finally create a model view projection matrix to render video on screen.

You can get full ViewRenderer.java from here.

Using ViewRenderer.java in your Activity

Create object of VideoRenderer.

private final VideoRenderer videoRenderer = new VideoRenderer()

call videoRenderer.createOnGlThread(); inside onSurfaceCreated(GL10 gl,EGLConfig config).

In onDrawFrame(GL10 gl) method create a movie anchor.

if (mMovieAnchor == null) {
    mMovieAnchor = hit.createAnchor();
} else {
    anchors.add(hit.createAnchor());
}

at the end of onDrawFrame() method you can use this code to update video pose.

if (mMovieAnchor != null) {
    if (!videoRenderer.isStarted()) {
        videoRenderer.play("video_name.mp4", this);
    }
    mMovieAnchor.getPose().toMatrix(mAnchorMatrix, 0);
    videoRenderer.update(mAnchorMatrix);
    videoRenderer.draw(viewmtx, projmtx);
}

You can find modified HelloARActivity.java from here