How to develop and build your first AI PC app on Intel NPU (Intel AI Boost)

You just received your new AI PC today and are wondering how to run hardware-accelerated AI apps on it. Here is your first Hello World try.

A few prerequisites:

First, you can check out https://pypi.org/project/openvino/, and download the latest version of OpenVINO 2024.0. It comes with NPU plugin by default.

Installing OpenVINO 2024.0 with PyPI and seeing the NPU plugin.

Additionally, you can install OpenVINO 2023.2 or 2023.3 LTS from the archive using the zipped file in an offline manner. These packages enable you to develop on NPU, CPU, GPU, and even the GNA in Intel Core Ultra processor.

It’s important to note that not *all* models can be integrated with NPU, due to certain limitations of architectures such as dynamic shape. However, there’s good news! When exploring VPU and Movidius, you’ll find that the models we previously converted have a high likelihood of working well. I tested the pose estimation, object detection (YOLOv8), and image classification demos from notebooks, and I’m thrilled to report that they work well, there are dozens of models that will work just out-of-the-box too if you try our notebooks. More importantly, if NPU is not supported, you can still develop the app on CPU and GPUs and still get great performance.

Here are the key steps to make it work:

1. Download and install pre-req on OpenVINO Notebooks from the installation instruction here: OpenVINO Windows Installation, and start the Jupyter.

Notebooks got you 100+ demos to try, but again only a selective number of them works on NPU, but you can try them on CPU and GPUs!

2. To use NPU for inference, change the device name to “NPU”, and run. So far I have tested the image classification and human pose estimation notebooks.

Select NPU as the inference device. :)

Image Classification Result! :)

And human pose estimation running on NPU.

Utilizing the NPU for low power inference.

Real-time demo :)

That’s it. Now you have hundreds of examples to play with including Stable Diffusions, LLM chatbots, object detection, and much more. Not all of them will fit the NPUs, but you still have a rather powerful iGPU, and CPU for the tasks in case you want to get most of them from your AI PC.

So many hardware accelerators you can use with Intel Core Ultra.

Want to learn more about the Neural Processing Unit (NPU)? Fun fact, it was called Intel Movidius VPU and you find lots of information about them. For example, here is a “Hello NPU” notebook you can read and learn more about how to use NPU for the first time too.

openvino_notebooks/notebooks/hello-npu/hello-npu.ipynb at latest ·…

Intel team is constantly updating and improving the support of NPU, so it would be great if you could also report and add in your requests here too. https://github.com/openvinotoolkit/openvino/issues

#iamintel