Easily Deploy and Monitor Computer Vision Pipelines Anywhere

See how you can apply this to many use cases beyond retail.

Photo by Matt Noble on Unsplash

Presented by Antonio Martinez

AI and computer vision (CV) applications have gained remarkable popularity in the retail world, driven by the need for retailers to enhance the customer experience, optimize operations, and stay competitive in a rapidly evolving market. Successfully deploying a CV initiative at the edge requires a deep understanding of the underlying hardware and how it affects AI model behavior.

The automated self-checkout open source project developed by Intel provides a framework to quickly deploy AI and CV workloads at the edge. The framework gives developers the power to easily launch AI pipelines and use telemetry to monitor performance across the Intel® portfolio of heterogeneous architectures and hardware configurations.

In this post, we’ll show you how to deploy and monitor your AI workloads, how to easily move your workloads from a CPU to a GPU, and how to apply the self-checkout reference implementation to many use cases beyond retail.

Open source software stack for telemetry

Leveraging open source software like InfluxDB* and Telegraf* in concert with Intel® hardware offers a powerful solution for creating telemetry dashboards to monitor AI and CV workloads at the edge.

• InfluxDB is a high-performance, open source time-series database designed for handling large volumes of time-stamped data. It serves as the storage back end for telemetry data, enabling efficient data retrieval and visualization.
• Telegraf is an agent-based data collection tool that gathers data from various sources and sends it to InfluxDB. It supports a wide range of input plugins, making it versatile for collecting data from sensors, cameras, and other sources associated with AI and CV applications at the edge.

As part of our automated self-checkout project, we automate the setup for InfluxDB and Telegraf, giving you greater control over metrics such as CPU, GPU, and memory.

Demo: How to monitor your AI workloads at the edge

First, begin running your open source software stack for InfluxDB and Telegraf by following these steps.

Once InfluxDB and Telegraf are running and the dashboard has been imported, your dashboard will display gauges that monitor system performance across your AI workloads. In this example, our dashboard is monitoring an Intel® integrated GPU (iGPU), an Intel® Arc™ GPU, a CPU, and system memory.

An example telemetry dashboard

Running your AI workload

Next, we’ll deploy our workload. Using the GitHub* quick-start guide, follow steps one through three to build the project, download the sample videos, and run the camera simulator.

Then, complete step four to run the object detection workload using the SSD MobileNet v1 model.

PIPELINE_PROFILE="object_detection" RENDER_MODE=1 sudo -E ./run.sh --workload ovms --platform dgpu.0 --inputsrc rtsp://127.0.0.1:8554/camera_0

By default, the AI workload will run on your CPU. This demo uses sample videos that you can find here.

Once you’ve run the command above, your inferencing results will look like this:

Example inference results.

As shown here, the model applies bounding boxes around the objects in the video, correctly identifying them as bottles and providing the accuracy percentages of the model’s predictions. The latency and frames per second (FPS) can be found in the top left corner.

Additional data is collected in the results folder. In this case, if you open the “r0.jsonl” file, you’ll find the following output:

• Processing time: 22.35 ms; fps: 20.22
• Processing time: 22.34 ms; fps: 20.23
• Processing time: 22.34 ms; fps: 20.23

Moving your workload from a CPU to a GPU

We’ll now be running the same workload on an iGPU and monitoring its performance. To do so, install Intel® GPU Tools — a collection of open source tools for developing and testing Intel® DRM drivers — to monitor GPU processes. On Ubuntu*, execute the following command:

$ sudo apt-get install -y intel-gpu-tools 

Once the toolkit is installed, use the command sudo intel_gpu_top to see the GPU utilization.

An example Intel® GPU Tools dashboard monitoring GPU utilization.

Notice only two processes — gnome-shell and system — are using the GPU at this point. You can also use this tool to visualize the overall render/3D value on the InfluxDB dashboard.

Now, let’s run a second pipeline, switching from CPU to GPU, to demonstrate how easy it is. For that, use this command:

DEVICE=GPU PIPELINE_PROFILE="object_detection" RENDER_MODE=1 sudo -E ./run.sh --workload ovms --platform dgpu.0 --inputsrc rtsp://127.0.0.1:8554/camera_4

For this workload, we’re running an input video of a traffic camera to demonstrate how this process can be applied to more workloads than self-checkout use cases. In this use case, the model identifies vehicles as they pass through an intersection.

A computer vision‒enabled traffic camera identifies passing vehicles.

When we review the intel_gpu_top utility, it shows that two processes of OpenVINO™ Model Server (ovms)are running on the GPU. We can see that utilization for the GPU render/3D is around 75 percent.

An example Intel® GPU Tools dashboard monitoring GPU utilization from a traffic camera workload.

Using these tools, you can launch and monitor even more AI workloads to help optimize performance. Use command make list-profiles to show the available pipelines and replace the value of PIPELINE_PROFILE with the workload you want to deploy.

Get started with the CV reference implementation

The automated self-checkout project equipped with telemetry using InfluxDB and Telegraf represents a leap forward in monitoring and analysis of AI workloads on various Intel® hardware components, including CPUs and GPUs. Explore how this innovative approach can give you a comprehensive view of your AI and CV application performance at the edge.

· Learn more about the AI-enabled CV checkout framework

· Read more about the automated self-checkout reference implementation

About the author

Antonio is a software engineer with a strong focus on AI applications. He has more than 10 years of experience in software development. He currently holds the esteemed position of technical lead at Intel, where he is responsible for guiding complex projects and leading teams to develop cutting-edge AI solutions. He has a bachelor’s degree in computer engineering and a master’s in computer science. Find him on LinkedIn and GitHub.