Reading Industrial Meters with OpenVINO™

Meter reading is an extremely important task that is widely used in real life. Many meter readings require manual periodic inspection and recording of the meter readings to reflect whether the device is operating safely. Due to the various shapes, scales, pointers, and characters of the meter, and the existence of different equipment locations, such as high-voltage equipment in substations, it is difficult for regular manual inspections to be carried out.

At present, traditional meters with scales and pointers are still commonly used in the substation environment. In the actual application environment, traditional manual counting not only involves many human factors but is also easily disturbed by the environment and has the danger of an electric shock. With the gradual promotion of unattended substations, inspection robots or UAVs equipped with automatic meter identification technology have been widely used. Therefore, the realization of intelligent reading of meters by the computer vision method has become a research hotspot [1].

Hence, in this article, we will see a Deep learning-based solution to read industrial meters that was created under Google Summer of Code 2023 and is implemented under the auspices of Intel’s OpenVINO Toolkit organization.

Reading meters with DL solution

Github Repo: ashish-2005/Industrial Meter Reading With OpenVINO

Part 1: Approach

The main task is to “read every meter in a frame/image with accuracy”. The whole task can be classified as a Computer vision task, as all the data we will receive will be in the form of images or videos.

The task can further be classified into two major tasks:

• Detection
• Segmentation

Detection

First and foremost, we need to detect the meter in the frame, and there might be one or more meters present in a single frame. Detection is one of the most basic computer vision tasks, and there are a lot of approaches to tackle this. For this solution, we have chosen Deep Learning because of the immense research and support available for object detection with DL.

In Deep Learning as well, there are a lot of models and architectures catering to the same purpose of object detection; it’s always hard to pick one over the other. However, we have chosen EfficientDet-d0 for detection, which is a scalable and efficient object detection model [2] and achieves state-of-the-art precision with fewer parameters.

Detection Model Result

Segmentation

Once we detected the meter in the frame and cropped it, the next step was to find the scale and pointer and then use this information to calculate the reading. For this, we used the approach of semantic segmentation, which means we are going to classify every pixel in the image as one of the classes, which will lead to a segmentation map that can be used to locate pointers and scale, hence calculating reading. Again for this, there are many models like UNET, DeepLab, FCN, etc. Out of UNET and DeepLab, we chose UNET as the difference in inference time and accuracy is minimal [3]

Segmentation Model Result

Calculating The Reading

Once we get the segmentation map, the last thing to do is calculate readings with the segmentation map. For this, we first convert the circular meter segmentation to a rectangle, then find the pointer location and scale location, and with the help of their relative locations, we can finally calculate readings according to the meter range and meter intervals.

Calculating Reading with Semantic-Segmentation Map

Implementation

To implement the solution, we have used TensorFlow as a Deep Learning framework, along with some other libraries like the TensorFlow official Object detection API and SegmentationModel. For inference, we have used the OpenVINO framework.

Detector Training

We used EfficientDet from Tensorflow’s model garden and trained it with the official TF object detection API. It is an open-source project that provides all the tools required to build and train a model from scratch in TensorFlow. You can choose from a variety of model architectures available on TF’s model zoo.

You can follow the training code and steps in detector_training.ipynb .

Segmenter Training

We have used UNET from Segmentation-Model. Segmentation-Model is an open-source Python library that contains TensorFlow implementations of the most famous segmentation models like UNET, FPN, PSPNet, and LinkNet.

You can follow the training code and steps in segmentor_training.ipynb .

Since most meters have different shapes, sizes, and features, it’s recommended to re-train the model with your own dataset featuring your type of meter.
To train on your custom data, all you need are two directories containing images and segmentation maps or annotation maps separately. Then convert them to data with the `load_image` function. The Rest of the code and steps remain the same as per the above-mentioned notebook.

OpenVINO Inference

Once we have the trained model, all we have to do is build a pipeline that can take a frame as input and return another image with the plotted result.
We can directly use the TensorFlow models, but we have used the OpenVINO runtime for inference because of the range of tools available that can help us improve inference and make deployment easier.
To use OpenVINO inference, we will need our model to be serialized in OpenVINO Intermediate representation, i.e., an XML and BIN files, which is a standardized file format for representing neural network models optimized for use with Intel hardware. These OpenVINO IR files make the solution cross-platform, as they can be used on any system with the OpenVINO Runtime.

After getting OpenVINO IR we can use the models in our pipeline to achieve desired results.

def pipeline(frame):
    '''
    Complete pipeline
    Image --> Detection --> Cropping --> Semantic-Segmentation --> Reading --> Plotting

    Parameters
        frame (np.ndarray) : image

    Retursn
        plotted_image (np.ndarray) : image with detection box and reading
    '''


    # Getting Detection BBOXs
    frame = cv2.resize(frame,(1920,1080),cv2.INTER_AREA)
    prediction = detector(frame.reshape(1,1080,1920,3))
    result_ratio = prediction['detection_boxes'][0]
    scores = prediction['detection_scores'][0]
    selected_rr = []
    
    # Using detection above threshold confidence
    for i in range(len(scores)):
        if scores[i]>0.5:
            selected_rr.append(result_ratio[i])
        else:
            break
    try:
        # Converting BBOX ratios to pixels 
        results = np.multiply(selected_rr,[1082,1920,1082,1920]).astype(np.int64)
    except:
        return(frame)

    # Cropping Meters
    roi_imgs,loc = roi_crop(frame,results,1,1)

    # Preprocess uneven cropped imgs to 256,256
    crop_img = []
    for roi_img in roi_imgs:
        resized = cv2.resize(roi_img,(256,256),cv2.INTER_AREA)
        crop_img.append(resized)

    if len(crop_img) > 0:
        # Getting Segmentation Maps
        pred_seg = segmentor(np.array(crop_img))[segmentor.output(0)]
        processed_seg = []
        for i in pred_seg:
            processed_seg.append(cv2.resize(i,(512,512),cv2.INTER_AREA))
        pred = np.argmax(processed_seg,axis=3)

        # Getting Reading from predicted Maps
        pred = erode(pred,2)
        rectangle_meters = circle_to_rectangle(pred)
        line_scales, line_pointers = rectangle_to_line(rectangle_meters)
        binaried_scales = mean_binarization(line_scales)
        binaried_pointers = mean_binarization(line_pointers)
        scale_locations = locate_scale(binaried_scales)
        pointer_locations = locate_pointer(binaried_pointers)
        pointed_scales = get_relative_location(scale_locations, pointer_locations)
        meter_readings = calculate_reading(pointed_scales)

        # Plotting reading and BBOXs on image
        plotted_img = plot_result(frame,meter_readings,results)

    else:
        plotted_img = frame
    return plotted_img

The pipeline will first preprocess the image for detection, then crop the detected part and preprocess it for segmentation, then use the predicted segmentation map for calculating readings, and finally plot the results. For more details on the code and inference process, check out OV-meter-reader.ipynb .

Part 2: Optimization and GUI

Since now we have a DL solution to read analog meters, the next step is to optimize the pipeline for multi-camera input along with a user-friendly web-based GUI.

Optimization

The aim of the project is to monitor multiple analog meters simultaneously, so we can’t just rely on step-by-step or synchronous execution, as until an inference is completed, the result might be stale. So to overcome this issue, we use asynchronous pipeline execution.

Asynchronous Inference with OpenVINO™

OpenVINO Runtime supports inference in either synchronous or asynchronous mode. The key advantage of the Async API is that when a device is busy with inference, the application can perform other tasks in parallel (for example, populating inputs or scheduling other requests) rather than wait for the current inference to complete first.

AsyncInferQueue: Asynchronous mode pipelines can be supported with the AsyncInferQueue wrapper class. This class automatically spawns the pool of InferRequest objects (also called “jobs”) and provides synchronization mechanisms to control the flow of the pipeline. It is a simpler way to manage the infer request queue in Asynchronous mode.

We achieved this by creating a pipeline, which follows:

1. The `main_loop` will capture or read a frame and pre-process it for detection, hence creating a detection InferRequest.
2. Once this detection InferRequest is complete, it will call the `detector_callback` function, which will further process the detection result and start a segmentation InferRequst.
3. And at last, once the segmentation InferRequest is complete, and it will call the `segmentor_callback` function, which will calculate reading and plot results.

High-level Performance Hints

Even though all supported devices in OpenVINO™ offer low-level performance settings, utilizing them is not recommended outside of very few cases. The preferred way to configure performance in the OpenVINO Runtime is using performance hints. This is a future-proof solution fully compatible with the automatic device selection inference mode and designed with portability in mind.

Throughput & Latency: Both “THROUGHPUT” and “LATENCY” performance hints automatically configure the asynchronous pipeline to use the optimal number of streams and inference requests.

Multi-Stream Execution: Multiple streams are created for the model. In the case of the CPU plugin, each stream has its own host thread, which means that incoming infer requests can be processed simultaneously.

Results
After trying different combinations of these optimizations, we decided to go with Performance Hint: Latency along with Multi-Stream execution.

Web-based GUI

To make our project a bit more of a real-world solution, we decided to create a web-based GUI with which users can monitor multiple meters with our DL solution.

When it comes to the backend for a website, Flask and Django turn out to be the best choices because of their flexibility, features, easy implementation, and immense amount of support available. For this project, we went with Flask because it's relatively simpler and lighter than Django.

For the frontend, we used HTML, CSS, and JavaScript to make it more user-friendly, add some important functionality, and act like a bridge to process backend-generated data.

Home Page
The home page is basically where everything will be happening, multi-meter monitoring, the latest three readings for each meter will be displayed along with a bounding box plotted image. Hence, the user can monitor multiple meters through a live video source.

Settings Page
On the settings page, the user can modify meter parameters like the range of the meter, the number of intervals, and the refresh time. Refresh time is the time after which the camera will capture a new frame, process it, and display the output.
We want to empower the user with as much flexibility as we can, so instead of hardcoding these parameters, we let the user decide. Hence, the user can monitor multiple different meters simultaneously.

Conclusion

We have successfully created and implemented a deep learning solution to read industrial meters with the help of OpenVINO toolkit. The overall solution could be easily optimized and deployed on different devices, such as CPU, iGPU, etc. with OpenVINO. It could also be extended to read different types of industrial meters with a few configurations. To bring our solution one step closer to the real-world solution, We have further optimized the solution pipeline to support multi-camera input. A web-based GUI has also been created to make it more user friendly.

References

[1] Automatic Meter Reading from UAV Inspection Photos in the Substation by Combining YOLOv5s and DeeplabV3+, 2022.

[2] EfficientDet: Scalable and Efficient Object Detection (2020)

[3] Comparison of Deep-Learning-Based Segmentation Models: Using Top View Person Images (2020)