Creating a high-performance React Native Vision Camera C++ frame processor using JSI

Setting up and using functions in C++ to handle the frame processor in the React Native app for Android and iOS platforms.

Photo by Ryan James Christopher on Unsplash

In the new version of React Native Vision Camera, a brilliant library for handling the camera in React Native applications, we can use the mechanism of customised frame processors that allow us to process frames in languages native to the device. This time, we will create a frame processor together that uses functions in C++ using JSI. This has the advantage of increasing performance due to the nature of C++, which allows much faster memory referencing at a lower level. An additional advantage is that a single code can be created to run on different platforms, which saves time.

The detailed way to create a frame processor for the Vison Camera library for the Android (Java) and iOS (Objective-C) platforms is described in my previous articles:

Creating native frame processors for Vision Camera in React Native using OpenCV

Real-time pose detection in React Native using MLKit

Due to the level of complexity, this article will only focus on the creation of the processor along with a description of the mechanisms behind this. I am writing this article when the current version of React Native is 0.73.5 and Vision Camera is 3.9.0.

Table of contents:

1. Configuration of the project and required libraries
2. Android library configuration
3. iOS library configuration
4. C++ module
5. Integration of the module into a React Native application
6. Summary

Configuration of the project and required libraries

In order to use the library from JSI, we will create an external library using the React Native Builder Bob. This library will enable us to quickly generate code. To do this, let’s use the command:

npx create-react-native-library@latest vision-jsi-processor

Next, we need to go through the library creation wizard. The most important information here is to select the library type as Turbo module with backward compat and C++for Android & iOS.

Builder Bob library configuration screen.

After entering the projects directory, let’s execute the command yarn.

Warning. If you experience installation problems, create the yarn.lock file with the command touch yarn.lock , and then repeat yarn.

Vision Camera configuration

The next step will be to install the necessary libraries. After react-native-vision-camera you will need to use react-native-worklets-core (to be able to use native frame processors).
In the package.json of the library add:

In the devDependencies section:

 "react-native-vision-camera": "3.9.0"

In the peerDependencies section:

"react-native-vision-camera": "*",
"react-native-worklets-core": "*"

We will test our library in the example generated by Bob. So let’s also add the necessary libraries in the example/package.json file (dependencies section):

"react-native-vision-camera": "^3.9.0",
"react-native-worklets-core": "^0.3.0"

Then, in the root directory of the library, let’s run the yarn command again.

The next steps are to add the necessary permissions to the camera. In the Info.plist file in the example/ios directory, let’s add the following entry:

<key>NSCameraUsageDescription</key>
<string>$(PRODUCT_NAME) needs access to your Camera.</string>

Then in the file example/android/app/src/main/AndroidManifest.xml:

<uses-permission android:name="android.permission.CAMERA" />

The last thing to do is to configure the worklets-core library support by adding an entry in the file example/babel.config.js

module.exports = {
  plugins: [
    ["react-native-worklets-core/plugin"],
    // ... 
  ],
  // ... 
};

Android library configuration

The first step in this section will be to configure the required modules. As we will be using C++ it is necessary to configure the android/CMakeList.txt file to the following form:

cmake_minimum_required(VERSION 3.4.1)
project(Ocv)

set (CMAKE_VERBOSE_MAKEFILE ON)
set (CMAKE_CXX_STANDARD 14)

find_package(ReactAndroid REQUIRED CONFIG)
find_package(fbjni REQUIRED CONFIG)
find_package(react-native-vision-camera REQUIRED CONFIG)

add_library(react-native-vision-jsi-processor            SHARED
            ../node_modules/react-native/ReactCommon/jsi/jsi/jsi.cpp
            ../cpp/react-native-vision-jsi-processor.cpp
            ../cpp/react-native-vision-jsi-processor.h
            cpp-adapter.cpp
)

# Specifies a path to native header files.
include_directories(
            ../cpp
            ../node_modules/react-native/React
            ../node_modules/react-native/React/Base
            ../node_modules/react-native/ReactCommon/jsi
            ../node_modules/react-native-vision-camera/android/src/main/cpp/frameprocessor
            ../node_modules/react-native-vision-camera/android/src/main/cpp/frameprocessor/java-bindings
)

target_link_libraries(
        react-native-vision-jsi-processor
        android  
        ReactAndroid::jsi
        fbjni::fbjni
        react-native-vision-camera::VisionCamera
)

Explanation:

• The add_library section contains all the files of our library.
• The include_directories section are the directories that contain the files required for compilation, the dependencies relating to JSI and Vision Camera.
• The target_link_libraries section is the entire libraries that are also required.

The Vision Camera library is built from so-called prefabricated files, which enable us to use them in our own native code. Hence, it is necessary to customise our library still in the android/build.gradle file:

def reactNativeArchitectures() {
  def value = project.getProperties().get("reactNativeArchitectures")
  return value ? value.split(",") : ["armeabi-v7a", "x86", "x86_64", "arm64-v8a"]
}

android {
  //...

  buildFeatures {
    prefab true
  }

 

  defaultConfig {
    //...
    externalNativeBuild {
      cmake {
        cppFlags "-O2 -frtti -fexceptions -Wall -fstack-protector-all"
        abiFilters (*reactNativeArchitectures())
        arguments "-DANDROID_STL=c++_shared"
      }
    } 
  }

  packagingOptions {
    excludes = [
      "META-INF",
      "META-INF/**",
      "**/libc++_shared.so",
      "**/libfbjni.so",
      "**/libjsi.so",
      "**/libreactnativejni.so",
      "**/libturbomodulejsijni.so",
      "**/libreact_nativemodule_core.so",
    ]
  }

  externalNativeBuild {
    cmake {
      path "CMakeLists.txt"
    }
  }

  //...
}

The next step will be to create a function that will add the ability for us to use the native method in C++ in our Javascript code. To do this, we need to change the contents of the android/src/main/java/com/visionjsipr
ocessor/VisionJsiProcessorModule.java file, to the following:

package com.visionjsiprocessor;

import android.util.Log;

import androidx.annotation.NonNull;

import com.facebook.react.bridge.JavaScriptContextHolder;
import com.facebook.react.bridge.Promise;
import com.facebook.react.bridge.ReactApplicationContext;
import com.facebook.react.bridge.ReactContextBaseJavaModule;
import com.facebook.react.bridge.ReactMethod;

public class VisionJsiProcessorModule extends ReactContextBaseJavaModule {
  public static final String NAME = "VisionJsiProcessor";

  VisionJsiProcessorModule(ReactApplicationContext context) {
    super(context);
  }

  @Override
  @NonNull
  public String getName() {
    return NAME;
  }

  static {
    System.loadLibrary("react-native-vision-jsi-processor");
  }

  public static native void nativeInstall(long jsi);

  @ReactMethod(isBlockingSynchronousMethod = true)
  public void install() {
    JavaScriptContextHolder jsContext = getReactApplicationContext().getJavaScriptContextHolder();

    if(jsContext.get() != 0) {
      nativeInstall(jsContext.get());
    } else {
      Log.e("SimpleJsiModule", "JSI Runtime is not available in debug mode");
    }
  }
}

The install method is a function that can be called directly from the Javascript code. It serves us to so-called install the module, i.e. to call the necessary code written in C++, i.e. the nativeInstall method .
The implementation of this method must be found in the android/cpp-adapter.cpp file:

#include <jni.h>
#include <jsi/jsi.h>
#include "react-native-vision-jsi-processor.h"

using namespace facebook;

extern "C"
JNIEXPORT void JNICALL
Java_com_visionjsiprocessor_VisionJsiProcessorModule_nativeInstall(JNIEnv *env, jclass type, jlong jsi) {
    auto runtime = reinterpret_cast<facebook::jsi::Runtime *>(jsi);

    if (runtime) {
        visionjsiprocessor::install(*runtime);
    }
}

This function allows us to call another install function also written in C++. Its implementation will be addressed in a moment.

iOS library configuration

The next step will be to prepare the library on the iOS side. Let’s replace the contents of the ios/VisionJsiProcessor.mm file:

#import "VisionJsiProcessor.h"
#import <React/RCTBridge+Private.h>
#import <React/RCTUtils.h>
#import <jsi/jsi.h>
#import "react-native-vision-jsi-processor.h"

@implementation VisionJsiProcessor

@synthesize bridge = _bridge;
@synthesize methodQueue = _methodQueue;

RCT_EXPORT_MODULE()

+ (BOOL)requiresMainQueueSetup {

    return YES;
}

- (void)setBridge:(RCTBridge *)bridge {
    _bridge = bridge;
}

RCT_EXPORT_BLOCKING_SYNCHRONOUS_METHOD(install)
{
    RCTCxxBridge *cxxBridge = (RCTCxxBridge *)self.bridge;
    if (!cxxBridge.runtime) {
        return @false;
    }

    visionjsiprocessor::install(*(facebook::jsi::Runtime *)cxxBridge.runtime);
    return @true;
}

// Don't compile this code when we build for the old architecture.
#ifdef RCT_NEW_ARCH_ENABLED
- (std::shared_ptr<facebook::react::TurboModule>)getTurboModule:
    (const facebook::react::ObjCTurboModule::InitParams &)params
{
    return std::make_shared<facebook::react::NativeVisionJsiProcessorSpecJSI>(params);
}
#endif

@end

This code, similar to what we did earlier for Android, allows us to create an install function that will call another install function, but whose implementation is in the cpp file. After any changes, don’t forget to call the command under install in the example/iOS directory.

C++ module

Once the module in both pages has been added correctly, it will be necessary to add the implementation of the install function in the language source file C++ Let’s start with the header file cpp/react-native-vision-jsi-processor.h:

#ifndef VISIONJSIPROCESSOR_H
#define VISIONJSIPROCESSOR_H

#include <jsi/jsilib.h>
#include <jsi/jsi.h>

namespace visionjsiprocessor {
  void install(facebook::jsi::Runtime& jsiRuntime);
}

#endif /* VISIONJSIPROCESSOR_H */

The sole purpose here is to create a function definition. Then its implementation should be found in the file cpp/react-native-vision- jsi.process.cpp:

#include "react-native-vision-jsi-processor.h"
#include <jsi/jsi.h>
#ifdef ANDROID
#include "../../../node_modules/react-native-vision-camera/android/src/main/cpp/frameprocessor/FrameHostObject.h"

using namespace vision;
#else
#include "../../../node_modules/react-native-vision-camera/ios/Frame Processor/FrameHostObject.h"
#endif

namespace visionjsiprocessor {
  using namespace facebook;



    void install(jsi::Runtime& runtime) {
        auto myPlugin = [=](jsi::Runtime& runtime,
                            const jsi::Value& thisArg,
                            const jsi::Value* args,
                            size_t count) -> jsi::Value {
            auto valueAsObject = args[0].getObject(runtime);
            auto frame = std::static_pointer_cast<FrameHostObject>(valueAsObject.getHostObject(runtime));

            int test;
#ifdef ANDROID
            test = frame->frame->getHeight();
#else
            test = frame->frame.height;
#endif
            return jsi::Value((int)test);
        };

        auto jsiFunc = jsi::Function::createFromHostFunction(runtime,
                                                             jsi::PropNameID::forUtf8(runtime,
                                                                                      "frameProcessor"),
                                                             1,
                                                             myPlugin);

        runtime.global().setProperty(runtime, "frameProcessor", jsiFunc);
    }
}

Unfortunately, currently with the Vision Camera library, definitions vary from system to system, hence the necessary use of preprocessor directives. The purpose of the install function is to add a method called frameProcessor, which we will call in our Javascript code, to a global variable.

Integration of the module into a React Native application

The library is already configured on the native side. Let’s now move on to the Javascript implementation of the module. In the src/index.tsx file, let’s add the definition of the native module:

import { NativeModules, Platform } from 'react-native';

const LINKING_ERROR =
  `The package 'react-native-vision-jsi-processor' doesn't seem to be linked. Make sure: \n\n` +
  Platform.select({ ios: "- You have run 'pod install'\n", default: '' }) +
  '- You rebuilt the app after installing the package\n' +
  '- You are not using Expo Go\n';

const VisionJsiProcessorModule = NativeModules.VisionJsiProcessor;

const VisionJsiProcessor = VisionJsiProcessorModule
  ? VisionJsiProcessorModule
  : new Proxy(
      {},
      {
        get() {
          throw new Error(LINKING_ERROR);
        },
      }
    );

export function install() {
  VisionJsiProcessor.install();
}

Here we have the install function , which will enable us to use the native method of the same name. Let’s now test the operation using our solution in the application. Let’s add the following code in the example/src/App.tsx file:

import * as React from 'react';

import { StyleSheet, View, Text } from 'react-native';
import { install } from 'react-native-vision-jsi-processor';
import {
  Camera,
  useCameraPermission,
  useCameraDevice,
  useFrameProcessor,
} from 'react-native-vision-camera';

install();

export default function App() {
  const { hasPermission, requestPermission } = useCameraPermission();
  const device = useCameraDevice('back');

  React.useEffect(() => {
    if (!hasPermission) {
      requestPermission();
    }
  }, [hasPermission, requestPermission]);

  const frameProc = global.frameProcessor;

  const frameProcessor = useFrameProcessor(
    (frame) => {
      'worklet';
      console.log(frameProc(frame));
    },
    [frameProc]
  );

  if (device == null)
    return (
      <View style={styles.container}>
        <Text>No device</Text>
      </View>
    );
  return (
    <Camera
      style={StyleSheet.absoluteFill}
      device={device}
      isActive={true}
      frameProcessor={frameProcessor}
    />
  );
}

It allows us to use functions from the global object in the frame processor, which is then used in the Camera component. Our custom processor, all it does is display the frame height — the specific implementation is something we can add later. So let’s test its operation:

Result returned by the frame processor (frame height).

Summary

We can use the Frame processor prepared in this way for many optimal solutions, e.g. calls to machine learning models, image processing, e.g. via the OpenCV library, etc. This solution undoubtedly has a big advantage in terms of performance and the possibility of sharing code between platforms. On the other hand, the disadvantages at this point are the undoubted difficulties and entry threshold associated with setting up such a module and the numerous debugging problems.

The code from the article is also available on the repository: https://github.com/lukaszkurantdev/blog-vision-frame-processor-cpp

Also take an interest in my other articles:

Using biometrics in React Native

Creating a native Smart Replay module in React Native using MLKit

Using iCloud containers in a React Native application