FastOpenCV — How to easily use OpenCV in React Native (also with VisionCamera!) in 2024

Demonstration of a library for React Native using JSI and C++ API.

react-native-fast-opencv is available.

I am pleased to announce the release of a new react-native-fast-opencv library allowing the OpenCV library to be used directly in JavaScript code, without manual configuration.

The library is adapted to work with the VisionCamera library and WorkletsCore, but also without additional integrations. It uses the C++ API and also works with the new architecture.

The library is in the early stages of development and not all functions or objects available in OpenCV are supported.

Documentation of the library can be found here.

Example of use

I will now show an example use of the library for real-time detection of coloured cards.

Real-time detection of coloured cards.

Let’s prepare a new React Native project:

npx react-native init opencvtest

Now let’s install all the necessary libraries:

• react-native-fast-opencv
• react-native-vision-camera — to handle the camera.
• react-native-worklets-core — for handling the frame processors in VisionCamera
• vision-camera-resize-plugin — for scaling frames,
• @shopify/react-native-skia — for drawing objects on the frame screen.
• react-native-reanimated — dependency for frame processors.

yarn add  react-native-fast-opencv react-native-vision-camera react-native-worklets-core vision-camera-resize-plugin @shopify/react-native-skia react-native-reanimated

And then

cd ios && pod install

Configuration

Let’s move on to the configuration. In the Info.plist file, let’s add:

<key>NSCameraUsageDescription</key>
<string>$(PRODUCT_NAME) needs access to your Camera.</string>

And in the AndroidManifest.xml file:

<uses-permission android:name="android.permission.CAMERA" />

And in babel.config.js:

module.exports = {
  presets: ['module:@react-native/babel-preset'],
  plugins: [
    ['react-native-worklets-core/plugin'],
    'react-native-reanimated/plugin',
  ],
};

Implementation

Let’s start by creating a simple component that allows us to use the camera, asks for permissions and scales the frame for us:

const paint = Skia.Paint();
paint.setStyle(PaintStyle.Fill);
paint.setColor(Skia.Color('lime'));
 
export function VisionCameraExample() {
  const device = useCameraDevice('back');
  const { hasPermission, requestPermission } = useCameraPermission();
 
  const { resize } = useResizePlugin();
 
  useEffect(() => {
    requestPermission();
  }, [requestPermission]);
 
  const frameProcessor = useSkiaFrameProcessor((frame) => {
    'worklet';
 
    const height = frame.height / 4;
    const width = frame.width / 4;
 
    const resized = resize(frame, {
      scale: {
        width: width,
        height: height,
      },
      pixelFormat: 'bgr',
      dataType: 'uint8',
    });
 
  }, []);
 
  if (!hasPermission) {
    return <Text>No permission</Text>;
  }
 
  if (device == null) {
    return <Text>No device</Text>;
  }
 
  return (
    <Camera
      style={StyleSheet.absoluteFill}
      device={device}
      isActive={true}
      frameProcessor={frameProcessor}
    />
  );
}

We scale the frame to reduce its size and enable faster processing. In addition, we handle permissions for the Camera. Now let’s focus on the frame processor function. The goal is to detect green cards and mark them in the appropriate position on the screen. To enable the OpenCV library to process the image, we need to convert our frame to a Mat object:

const src = OpenCV.frameBufferToMat(height, width, resized);

Let’s also create another Mat object to store our processed photo.

const dst = OpenCV.createObject(ObjectType.Mat, 0, 0, DataTypes.CV_8U);

In order to find the object easily, it will be necessary to change the colour to HSV. We can also create objects (Scalar) that will be the beginning of our detected range and its end. The cvtColor function changes the colour format, while the inRange function leaves only those pixels whose colour fits within the specified range.

const lowerBound = OpenCV.createObject(ObjectType.Scalar, 30, 60, 60);
const upperBound = OpenCV.createObject(ObjectType.Scalar, 50, 255, 255);
OpenCV.invoke('cvtColor', src, dst, ColorConversionCodes.COLOR_BGR2HSV);
OpenCV.invoke('inRange', dst, lowerBound, upperBound, dst);

We further split the image into channels and extract the first channel.

const channels = OpenCV.createObject(ObjectType.MatVector);
OpenCV.invoke('split', dst, channels);
const grayChannel = OpenCV.copyObjectFromVector(channels, 0);

Now we will deal with finding the contours in order to do this we will use the findContours function.

const contours = OpenCV.createObject(ObjectType.MatVector);
OpenCV.invoke(
  'findContours',
  grayChannel,
  contours,
  RetrievalModes.RETR_TREE,
  ContourApproximationModes.CHAIN_APPROX_SIMPLE
);

Our detected card must be quite large to be detected. We therefore filter out those objects that are too small. To do this, we use the contourArea function to take the size of the contour and then, if it is larger than a fixed value, find a rectangle that will be able to cover it (boundingRect function).

for (let i = 0; i < contoursMats.array.length; i++) {
  const contour = OpenCV.copyObjectFromVector(contours, i);
  const { value: area } = OpenCV.invoke('contourArea', contour, false);
 
  if (area > 3000) {
    const rect = OpenCV.invoke('boundingRect', contour);
    rectangles.push(rect);
  }
}

We can mark the elements detected in this way using Skia.

frame.render();
 
for (const rect of rectangles) {
  const rectangle = OpenCV.toJSValue(rect);
 
  frame.drawRect(
    {
      height: rectangle.height * 4,
      width: rectangle.width * 4,
      x: rectangle.x * 4,
      y: rectangle.y * 4,
    },
    paint
  );
}

IMPORTANT. Remember to remove objects from the memory buffer at the end. Lack of this step, will result in continuous holding of values in memory — and consequently, in the case of frame processors, very fast filling of memory.

OpenCV.clearBuffers(); // REMEMBER TO CLEAN

Result

Cards are detected in real time when they are close enough to the lens.

Real-time detection of coloured cards

Summary

The react-native-fast-opencv library allows simple use of OpenCV and React Native. applications, and its use of JSI and C++ APIs ensures maximum performance. Simple integrations allow it to be used easily with VisionCamera or WorkletsCore.

You can find out more about the library in the official documentation here.

Also take an interest in my other articles:

High-performance hand landmark detection in React Native using Vision Camera and Skia frame…

Creating a high-performance React Native Vision Camera C++ frame processor using JSI

Creating native frame processors for Vision Camera in React Native using OpenCV