The battle to run my custom network on a Movidius / Myriad Compute Stick

Photo by Louis Reed on Unsplash

Intel Movidius

The Intel Movidius Neural Compute Stick 2 captured my attention as a cheap and seemingly simple possibility to run neural networks at the edge and do some serious AI inference on an edge device (for example a Raspberry Pi). With a price of around 70€ it’s really affordable. At my university more and more scientific staff have one lying around — not sure what to do with it. Since my department is not really doing vision nor NLP we didn’t want to port a standard model from the model zoo but have our own networks running on Movidius. So I set out to figure out how to do just that — turns out it’s not so straight forward. To make it easier for everybody else I decided to write this blog post. I hope it helps! The example code for the conversion of a simple NN from Scikit-Learn, Pytorch and Keras is available on my github.

Building and training your model

We wanted to be as generic as possible in our model training approach. After some despair when trying to port models directly from Tensorflow 2.0 using NCSDK similar to that of Oviyum, I found a rather simple way of transfer using the ONNX format. Using this we can build and train our network in PyTorch, Tensorflow, Keras or even SciKit Learn — well anything that’s able to export to ONNX. At their homepage they offer an overview of possible frameworks and functions. But not all functions are yet supported by Intel, a list of those can be found here. I plan on having a full converter at some point in the future, but up to now you have to check yourself if all your layers/functions are supported.

Installation

First up we need the Openvino Toolkit to convert our ONNX models to the Movidius format called Intermediate Representation (IR). I ran it on my Windows 10 and tested the examples using my Anaconda Shell — don’t forget to run the bash script that sets up the environment variables first every time you open a new shell! In addition you need python (Version 3 — I recommend Anaconda when running on Windows), an IDE of your choice (I prefer PyCharm) and some additional packages (best installed in a virtual environment). You need the framework you want to create your model in (Pytorch, Keras or SciKit Learn or something else), the ONNX package and maybe the converter that goes with your framework. For Keras conversion for example you need to run:

conda install scikit-learn, keras, pandas
pip install keras2onnx

Exporting your model

After you trained your model you have to export the model and the weights to ONNX format. It’s really simple. For your keras model you just do:

import keras2onnx
import onnx
onnx_model = keras2onnx.convert_keras(model, model.name)
onnx.save_model(onnx_model, 'keras_iris_model.onnx')

You can find code examples to save your model as onnx from keras, pytorch and sklearn on my github. If possible try to save the model without the training parameters.

Now let’s take a look at our model and check what parameters we saved.

Part of the network created using sklearn (visualized with Netron)

To open the onnx files I recommend Netron. It allows you to see all your layers, connections and even weights. Beautiful! But as you can see there’s more to our network than just the explicitly defined layers. Depending on your framework the model still has some training parameters or some convenience decoding parameters like this sklearn model. The Zip-Map here for example maps the output index to a concrete class name — nice to have, but impossible for Openvino model conversion.

Pruning your model

To successfully convert your model using Openvino you (sometimes) need to prune it. That means, you have to remove everything the model does not need or that is not available for Openvino conversion. One can easily remove layers or nodes, as they are called in ONNX, using the python onnx package.

import onnx
onnx_model = onnx.load(model_path)
graph = onnx_model.graph
graph.node.remove(graph.node[9])
onnx.save(onnx_model, save_path)

I know this is not the clean way to do it, but it still works. The long term goal here is a graph pruner, that removes all nodes that are not available for Openvino and throws an error if the model is not convertible. If you have something like this, don’t be shy and share it. Until then we need to do this by hand.

Converting your model to Openvino IR format

Now that we have our “perfect” ONNX model we can follow the “extremely informative” Intel tutorial on how to convert it. — It’s not very helpful… So instead rather follow those instructions:

1) Open a Shell (Anaconda) as an administrator
2) Activate the setupvars.bat
3) Make sure you installed the right prerequisites
4) Go to the Intel model optimizer directory (C:\Program Files (x86)\IntelSWTools\openvino\deployment_tools\model_optimizer\ )
5) Run the mo.py script providing the input shape and the model directory

python mo.py — input_model Path\to\your\model.onnx — input_shape [10,4] — log_level WARNING

6) See the SUCCESS message and have a little party
7) Copy the .xml and .bin files the model optimizer gave you to the directory of your choice

It took me some time to figure this out so here are the most common and cryptic errors I came about:

- Not all output shapes were inferred or fully defined for node “Add”. -> Means you need to set the input shape
- Cannot infer shapes or values for node “ZipMap”.
[ ERROR ] There is no registered “infer” function for node “ZipMap” with op = “ZipMap”. Please implement this function in the extensions. -> Well that means you need to prune your model since the ZipMap function is not available for Openvino
- PermissionError: [Errno 13] Permission denied: ‘Path’ -> you need to start the shell as an administrator

Running your model on the Movidius

Here again we can rely on the splendid documentation and user guides from Intel — NOT!

The only thing that helped me was to look at the examples that came with the toolkit under C:\Program Files (x86)\IntelSWTools\openvino\_2019.3.379\deployment\_tools\inference\_engine\samples\python\_samples — even though they do not contain many comments.
If you want to run the code in your IDE please make sure you activate the setupvars.bat before starting your IDE from the same shell (or any other way to do the setup). To run our network we need to import the two classes IECore and IENetwork. We load our model, define input and output blobs, load the network on the stick and are now finally ready to infer something! Again there’s a more in depth code on my github.

from openvino.inference_engine import IECore, IENetwork
ie = IECore()
net = IENetwork(model=model_xml, weights=model_bin)
input_blob = next(iter(net.inputs))
out_blob = next(iter(net.outputs))
exec_net = ie.load_network(network=net, device_name=’MYRIAD’)
res = exec_net.infer(inputs={input_blob: X_test[:10,:]})
res = res[out_blob]

I hope this helped you to get started with your Movidius Compute Stick for custom models and it saved your from the exasperation and despair with the Intel docs that I ran into. If you have some tips or improvements feel free to share!