3D Jupyter Notebooks and WebGL Integration for OpenVINO
Overview
This blog is about GSoC 3D Jupyter Notebooks and WebGL Integration for OpenVINO, built on OpenVINO Toolkit with Python.
OpenVINO Notebooks is a collection of ready-to-run Jupyter notebooks for learning and experimenting with the OpenVINO™ Toolkit. In machine learning, many of the tasks may be much better visualized and understood if we provide a 3D or 4D (space and time) perspective to the input video or images. So this project aims to provide the beginner-friendly Jupyter 3D engine for machine learning visualization along with WebGL integration.
Project Description
- You can find a more detailed description of the project here (GSOC Page).
- You can try running the demo here (binder).
As an attempt to integrate 3D display with OpenVINO, it can be implemented in many ways. In this project, I used the 3D human pose estimation model as a demonstration and used OpenCV and WebGL in two different ways to achieve the 3D display effects respectively. You can find more information about this model in the GitHub repository. The code works well on Windows and Ubuntu operating systems.
One goal for OpenVINO Notebooks is to reduce dependencies. So the consideration is to keep the same programming language and reduce dependency packages.
To integrate WebGL with OpenVino Notebooks, I create a few Python interfaces. An underlying implementation is a tool named Pythreejs, which uses Python API to call the Three.js library and WebGL. And since it works well on both Ubuntu and Windows operating systems, it does not require very complicated additional installation operations.
Another method is to use existing demo tools to map 3D graphics onto a 2D screen, which I did with OpenCV. In this demo, I defined some object classes, and matrix transformations, and drew them on the screen. You can find more information about Model, View, and Projection here if you forget what they mean. Other approaches may be tried in the future.
Code format specification and other requirements conforming to Contributing to OpenVINO Notebooks.
How is it working?
The overall process is to process each frame of the video, use the OpenVINO tool to infer, get the 3D coordinates after processing the results, and use the provided graphical interface to display the 3D images.
The following are the Python interfaces of the first method that are available to users.
# Attributes:
# view_width, view_height: Size of the view window.
# position: Position of the camera.
# lookAtPos: The point at which the camera looks at.
# axis_size: The size of axis.
# grid_length: grid size, length == width.
# grid_num: grid number.
# grid: If use grid.
# axis: If use axis.
engine3D = engine.Engine3js(view_width,
view_height,
position=[],
... )# define object
object = engine.draw_objects(object name, point size, points color)# add object in scene
engine3D.scene_add(...)# remove object in scene
engine3D.scene_remove(...)# draw object in scene
engine3D.plot()
When I use OpenCV to render such a 3D effect, I need to do some transformation of their position coordinates and map them to the 2D screen. Below is a code tree that shows the different parts of the graphical pipeline.
Visual3D
├── camera.py # Define camera parameter
├── cvEngine.py # Main entry
├── main.py
├── matrix_change.py # Some matrix calculation, such as translation
├── object_3d.py # Define the object, such as skeleton
└── projection.py # Define projection parameter
The 3D display notebook created could also be used to integrate the display of point cloud images, as shown in the following gifs. Actually, any new demo with 3D results using OpenVINO inferencing could be easily ported to this 3D engine notebook, showing its expansibility and reusability.
Conclusion
I would like to thank GSoC and the OpenVINO community for giving me this opportunity. I learned a lot of interesting and useful knowledge and skills from this open source project outside of my own major. Thanks to my 2 mentors for guiding me, as well as for regular weekly exchanges. What impressed me was that my mentors often encouraged me to be bold. The more I tried, the more I got. It was fun. It was an unforgettable experience and I look forward to contributing more to the OpenVINO Toolkit community in the future.
