Unity Advanced ML Agents and OpenVINO™ Toolkit Optimization

Abstract

This article will help us to use OpenVINO™ Toolkit with Reinforcement learning and create AI applications that can be consumed in any form. We will be using the Intel Deep Learning Toolkit from theOpenVINO™ toolkit SDK. For the simulation, we will be using the Unity IDE and for building Reinforcement learning walkthroughs we will be using the Unity ML agents.

Introduction

Here we are going to implement reinforcement learning using Unity ML Agents. Unity ML agents is an open source plugin offering from Unity where we can use the Reinforcement learning methods to simulate game scenes in unity. The implementation involves the following steps:

1. Downloading the GitHub repo from Unity itself and save it locally.
2. We will open the project in Unity IDE.
3. As the Reinforcement learning aspect is dealt with Tensorflow we download the Tensorflow C# plugin package for Unity and Add it our project.
4. We will then train the simulated game environment created in unity with the help of Intel Optimized python.
5. As the byte file is generated it is consumed in the Game as a facilitator to guide the machine learning process. As the generated files are built we will introduce OpenVINO™ Toolkit and with the help of OpenVINO™ Optimization process, we generate the *.xml and the *.bin file.

Getting Started

To work with the project we will be working on the separate folder of Unity ML agents as shown in the figure below.

The folder structure for Unity ML Agents

You can also check my detailed article on Getting Started with Unity ML Agents with Intel Optimized Python here.

After that, we will open up the Anaconda command prompt. Make sure we have an active internet connection. We will have to make a local copy of Unity ML Agents(v 0.5) from the GitHub so we clone the GitHub repo.

Unity-Technologies/ml-agents

Cloning the Repo

The main considerations over here are the prerequisites file. Inside the cloned repo and we have to search for the required file. For this, we will have to enable the anaconda environment and get inside it. We are powering the python with Intel optimized version.

Let’s go through the steps as shown below.

As per the details are given on the Unity website. We will have to set up the requirements that mean the essential files for setting up the environments. Hence the following command is to be entered:

(idp) C:\UnityMLAgentsAdvanced\ml-agents\ml-agents>pip install -r requirements.txt

Executing the requirement.txt file installs all necessary libraries as well as the supportive version of Tensorflow(1.8) too.

There can be some leftover files in Python that are also needed to be installed so we will enter this command to install the leftovers.

(idp) C:\ml-agents\ml-agents>pip install .

The steps shown above will set up the required environment to run the simulations.

Unity GYM

We will also configure Unity Gym( It is a supportive version of Open AI gym synched to Unity environment). The following command is to be entered:

(idp) C:\ml-agents\gym-unity>pip install .

Unity ML Agents Training

We are working with the latest version of Unity ML agents. The ways to access the project and modify accordingly needs to be covered as described below.

The basic steps are as follows.

1. Open the Unity ML agents project in Unity.

Explore different examples but we will be confined to the crawler example.

We add in the Reinforcement learning logic which has already been covered in my previous articles (https://software.intel.com/en-us/articles/get-started-with-unity-machine-learning-using-intel-distribution-for-python-part-1 https://software.intel.com/en-us/articles/game-dev-with-unity-ml-agents-and-intel-optimized-python-part-two ). Make sure when we build the project we choose the brain type to external. Then we will build the project. The .exe file generated will look like as shown below. We will train it.

The step now is the most important one. This step forms the basis of all the things we do in OpenVINO™ Toolkit. So for training the simulated model, we have to perform certain steps. As Unity ML agents are evolving we will cover the part where we pass the parameters for training and analyze a phase of the training process.

(idp) C:\ml-agents\UnitySDK>mlagents-learn C:\ml-agents\config\trainer_config.yaml — env=C:\ml-agents\UnitySDK\san — run-id=san — train

After 7hours of training and it is complete all in the CPU with Intel optimized python.

As the training process is complete we again open unity IDE and run the simulation with the trained model. The generated model looks like the way shown below.

Model Optimization with Intel OpenVINO™ toolkit

We will open the OpenVINO™ toolkit. There is tweak we need to perform, we need to change the extension of the bytes file generated after training from *.bytes to *.pb.again and it is one of the important step.

The files are saved in the models folder.

All the Tensorflow models are not supported we had to look for the closest match and also copy the pipeline.config file.

With that we used the OpenVINO™ toolkit Optimizer to generate the *.xml file and *.bin file which can be used for inference. The internal steps performed are stated below.

First of all, We have to select the model where it resides ( — input_model) followed by the custom operations for Tensorflow.

(ml-agents) C:\Intel\computer_vision_sdk_2018.3.343\deployment_tools\model_optimizer>python mo_tf.py — input_model C:\ml-agents\ml-agents\models\abhi-0\abhi_abhi-0.pb — tensorflow_use_custom_operations_config extensions/front/tf/mask_rcnn_support.json — tensorflow_object_detection_api_pipeline_config C:\ml-agents\ml-agents\models\abhi-0\pipeline.config — reverse_input_channels

The process is complete and we have successfully created the *.xml and *.bin file.

Accessing the pdb files

There are certain things we need to know when using OpenVINO™ Toolkit optimizer we have to pass the output nodes while we convert the frozen graph but we are unable to see the nodes at certain times. We then have to use Tensorboard to analyze the *.pdb model. The process is shown below and we will analyze the Reinforcement learning model for Unity ML agents.

We have to clone the main Tensorflow branch locally. We go inside the tensorflow folder.

C:\tensorflow\tensorflow\python\tools

The snapshot of the folder.

We will now use the tool which has import_pb_to_tensorboard file.Now we will analyze one of the pdb files generated

The command to generate the graph representation is shown below:

(ml-agents) C:\tensorflow\tensorflow\python\tools>python import_pb_to_tensorboard.py — model_dir C:\Users\abhic\Desktop\ml-agents\python\models\Spidey\raw_graph_def.pb — log_dir /tmp/tensorflow_manisha

Model Imported. Visualize by running: tensorboard — logdir=/tmp/tensorflow_manisha

Let us now initiate the tensorboard.

TensorBoard 1.7.0 at http://DESKTOP-QA7DS1F:6006 (Press CTRL+C to quit)

We will open the tensorboard in web browser posting the http link.

The graph structure will look like as shown below.

We will now expand the view.

We will double click to expand the view.

Keep a tab on the output node.

We check on the dense_3 graph and the outputs as shown below.

Now we have seen using Tensorboard we are able to find the output nodes. We can pass anyone as parameters as in order to get the model optimization done, you have to pass in an output value that you identify using the Tensorboard (such as /dense_4, /gradients, /Adam).

Summary

We have completed all the steps necessary to consume Unity ML agents with the optimized file generated from OpenVINO™ Toolkit and ready for inference.

In the article, we have shown how we can use the Unity ML pipeline to generate use cases with simulated models.

For the training process, we have used Intel Optimized Tensorflow which also allows us to run in the CPU too.

We have used OpenVINO™ Toolkit using the optimization method to generate the required *.xml and *.bin file.

We can share the files created using the IR to be consumed in the inference part so the application can be run on many Intel supported devices.