Why VPU’s are the best solution for IoT deep learning projects (with Pytorch)

Giavoni Giovanni
Oct 28 · 3 min read
Image for post
Image for post
Image by xresch from Pixabay

VPU (Visual Processing Unit) is a new piece of technology for processing visual data on IoT devices.

Nowadays, tech world offers a whole set of tools to serve your IoT devices with AI applications. For example you can find yourself thinking about buying an NVIDIA Jetson Nano or Coral TPU

There are soo many different approaches and different solutions.

But, while designing your project, bear in mind one simple piece of advice: whith IoT devices power consumption and efficiency are the leading requirements for your project to work and thermal concerns are not secondary.

And this is why VPU’s solutions may come in handy.

What VPU are made of

1. VLIW processors (Old technology revival)

VLIWs can achieve far higher performance, offering much more ILP with much lower silicon and power costs.

VLIW architecture was intoduced by J. A. Fisher back in 1983 in his paper “Very long instruction word architectures and the ELI-512,” (Proc. 10th Annu. Int. Symp. Computer Architecture, June 1983, pp. 140–150):

“More formally, VLIW architectures have the following properties:

There is one central control unit issuing a single long instruction per cycle. Each long instruction consists of many tightly coupled independent operations. Each operation requires a small, statically predictable number of cycles to execute. Operations can be pipelined.

In one of the lates article written by Fisher, Paolo Faraboschi and Cliff Young that you can find here:

https://www.researchgate.net/publication/224535028_VLIW_processors_once_blue_sky_now_commonplace

it is explained further how VLIW compiler works and why VLIW are name so:

VLIW’s compiler rearranges the program in advance, picking what to issue and when to issue it, in order to maximize the parallel execution while maintaining correct behavior. Other processors (called superscalar) rely on the hardware to do this while the program runs.

Since the VLIW compiler presents many operations to be issued at once, it usually is asked to bundle them into a single, very long, instruction word — hence the description VLIW (64 -1024 bit)

2. Homogenous On-Chip Memory

in VPU’s your data is stored on-chip in order to minimize latency and off-chip data transfer

The centralized on-chip memory architecture allows for up to 400 GB/sec of internal bandwidth

3. Vision accelerators

Vision accelerators are specialized processors designed to deliver high performance machine vision at ultra low power, without introducing additional compute overhead.

In other words there are processors entirely dedicated in processing your video frames.

4. Neural compute engine

dedicated hardware accelerator for running on-device deep neural network applications.

Are VPUs really efficient compared to GPU?

Do to the lack of documentation and research I will reference to one of the best study out there Exploring the Vision Processing Unit as Co-processor for Inference (https://core.ac.uk/download/pdf/185526545.pdf)

Using a pre-trained network model based on the GoogLeNet work by Szegedy et al. [3], we have observed that the performance during inference on a single VPU chip is only 4× slower in comparison with reference CPU and GPU implementations. By employing a multi-VPU configuration, however, we have demonstrated equivalent performance results. Yet, the expected thermal-design power (TDP) can still be reduced by a factor of 8×.

VPU machine learning framework

while Coral TPU is made by Google and only supports the Tensorflow framework, the Intel VPU (Neural Compute Stick 2 ) can support TensorFlow, Caffe, ApacheMXNet, Open Neural Network Exchange, PyTorch, and PaddlePadle via an Open Neural Network Exchange conversion.

If you want to know more about Intel VPU you can read this article published in IEEE Micro · March 2015:

https://www.researchgate.net/publication/275365424_Always-on_Vision_Processing_Unit_for_Mobile_Applications

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data…

Sign up for Analytics Vidhya News Bytes

By Analytics Vidhya

Latest news from Analytics Vidhya on our Hackathons and some of our best articles! Take a look

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Giavoni Giovanni

Written by

Change manager with a strong passion for coding

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data Science professionals. We are building the next-gen data science ecosystem https://www.analyticsvidhya.com

Giavoni Giovanni

Written by

Change manager with a strong passion for coding

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data Science professionals. We are building the next-gen data science ecosystem https://www.analyticsvidhya.com

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store