Object detection in the image using TensorFlow in NextJS

Object Recognition in Image

Object detection is a technique of identifying objects in images or videos using machine learning or deep learning algorithms. We are going to use TensorFlow.js which is a JavaScript library for machine learning. We will use coco-ssd, one of the models from TensorFlow which is used to classify and locate the object in an image. In this blog, we will dive into the process of implementing object detection in images using TensorFlow within a Next.js environment.

Prerequisites

1. TensorFlow.js
2. COCOSSD Model
3. NextJS

Before we move to implementation, please create and run NextJS application. You can find details on creating NextJS project here.

Implementation steps

1. Install required libraries
2. Implement and manage state for image upload feature
3. Load COCOSSD model
4. Use the COCOSSD method to scan the image and get the results
5. Draw image on canvas
6. Use results to draw a rectangle to highlight detected objects on the canvas

Install required libraries

Use the below commands to install TensorFlow.js and COCOSSD Model.

npm install @tensorflow/tfjs @tensorflow-models/coco-ssd

Implement and manage state for image upload feature

Create a file in the pages folder and copy the below code for image upload.

export default function ObjectDetection() {
    const canvasEle = useRef(null);
    const imageEle = useRef(null);
    const [uploadedImage, setUploadedImage] = useState(null);

    return (
        <div className={styles.container}>
            <div className={styles.imageSection}>
                <div className={styles.previewArea}>
                    {uploadedImage && (
                        <>
                            <Image
                                ref={imageEle}
                                src={uploadedImage}
                                alt='sample image'
                                width={500}
                                height={500}
                                style={{ objectFit: 'fill' }}
                            />
                            <canvas
                                ref={canvasEle}
                                className={styles.canvas}
                                width={500}
                                height={500}
                            />
                        </>
                    )}
                </div>
                <div>
                    <label htmlFor='fileSelect' className={styles.fileUpload}>
                        <span>
                            <i className='bi bi-upload'></i>
                        </span>
                        Upload an image
                    </label>
                    <input
                        id='fileSelect'
                        type='file'
                        onChange={setImage}
                        hidden
                    />
                </div>
            </div>
        </div>
    );
}

Load COCOSSD model

Import COCOSSD model in the component and use the load method to initialize the model, before we can use it to scan images.

Note: Make sure to place a loader in UI as it will take few seconds to load COCOSSD model.

import { load as cocoModalLoad } from '@tensorflow-models/coco-ssd';

export default function ObjectDetection() {
  const [objectDetector, setObjectDetectors] = useState(null);
  const [isLoading, setIsLoading] = useState(true);
  
  const loadOCRModel = async () => {
      const model = await cocoModalLoad();
      setObjectDetectors(model);
      setIsLoading(false);
  };

  useEffect(() => {
      loadOCRModel();
  }, []);
}

Use the COCOSSD method to scan the image and get the results

Use the initialized object of the model to detect objects from the image.
Also, create a button Start detection to trigger image detection.

export default function ObjectDetection() {
  const [detectedObjects, setDetectedObjects] = useState([]);
  
  const startDetecting = async () => {
    const image = browser.fromPixels(imageEle.current);
    const predictions = await objectDetector.detect(image);

    setDetectedObjects(predictions);
  };
}

Draw image on canvas

Before we highlight the detected objects in the image we have to draw the image on canvas.

    const draw = (ctx) => {
        canvasEle.current.width = imageEle.current.width;
        canvasEle.current.height = imageEle.current.height;
        
        ctx.fillStyle = '#000000';
        ctx.fillRect(0, 0, imageEle.current.width, imageEle.current.height);

        ctx.drawImage(
            imageEle.current,
            0,
            0,
            imageEle.current.width,
            imageEle.current.height
        );
    };

Here ctx param represents canvas 2d context, which is used to draw the image and shapes.

Use results to draw a rectangle to highlight detected objects on the canvas

In the last step, we will use the prediction results from the model and use the data to highlight detected objects on the canvas image.

const draw = (ctx, objects) => {
        canvasEle.current.width = imageEle.current.width;
        canvasEle.current.height = imageEle.current.height;
        // Clear part of the canvas
        ctx.fillStyle = '#000000';
        ctx.fillRect(0, 0, imageEle.current.width, imageEle.current.height);

        ctx.drawImage(
            imageEle.current,
            0,
            0,
            imageEle.current.width,
            imageEle.current.height
        );
        for (let i = 0; i < objects.length; i += 1) {
            // Draw the background rectangle for text
            ctx.fillStyle = 'rgba(0, 128, 0, 0.5)';
            ctx.strokeStyle = 'white';
            ctx.fillRect(
                objects[i].bbox[0],
                objects[i].bbox[1],
                objects[i].bbox[2],
                20
            );
            // Write image class on top left of rect
            ctx.font = '16px Arial';
            ctx.fillStyle = 'white';
            ctx.fillText(
                objects[i].class,
                objects[i].bbox[0] + 4,
                objects[i].bbox[1] + 16
            );

            // draw rectangle using data from prediction result
            ctx.beginPath();
            ctx.rect(
                objects[i].bbox[0],
                objects[i].bbox[1],
                objects[i].bbox[2],
                objects[i].bbox[3]
            );
            ctx.strokeStyle = 'green';
            ctx.stroke();
            ctx.closePath();
        }
    };

Conclusion

By following the above steps one can easily implement object detection using TensorFlow.JS and COCO SSD model in NextJS. Find the whole implementation with proper UI on my GitHub project demo given below. You can also check the live demo running on my website given below. Stay tuned for more technical blogs.

Github link:- https://github.com/NSV1991/nextjs-portfolio/blob/main/src/pages/work/object-detection/index.jsx

Live demo:- https://neerajvageele.com/work/object-detection