Getting hands dirty in Virtual Reality : Developing an Unreal Engine 4 Plugin for 5DT Data Gloves 14 Ultra

Source

In this blog, I am going to share my experiences from one of the most memorable projects in my AR/VR career to date. I was working as a Research Assistant in the Safety Analytics and Virtual Reality (SAVR) Lab at Indian Institute of Technology, Kharagpur back in 2017 and I was assigned the job of developing a UE4 plugin for a brand new pair of 5DT Data Gloves 14 Ultra. Nothing like that existed in the open source. So I had to build it almost from scratch. It was a tough but thoroughly enriching experience that brought me up close and personal with the core challenges in VR development. I believe the rest of the community will find my learnings useful. So in this blog, I am going to describe at length — how I approached the problem, the resources available at that time, how I figured out the missing pieces, built and delivered the plugin and finally, how people can replicate my procedure and make use of the plugin in their applications.

Source

Here is the problem statement that my Supervisor gave me:

“Develop an UE4 plugin for 5DT Data Gloves 14 Ultra which will serve as an interface for enabling reenactment of the user’s real hand movements on virtual hands in an Unreal Engine environment in order to manipulate virtual objects in the environment.”

After reading the problem statement, the following questions must be hitting your mind:

  1. What is Virtual Reality?
  2. What is Virtual World?
  3. What is Unreal Engine 4?
  4. What is Virtual Objects?
  5. What is 5DT Data Gloves 14 Ultra?

No worries! I will give you a brief on the essential concepts of Virtual Reality that should answer all these questions.

So, let’s begin…

What is Virtual Reality?

Here is how I understand and visualize Virtual Reality:

“Virtual Reality (VR) is a completely immersive experience that lands the user in a computer-generated 360 degrees environment in blink of her eyes. A Head-Mounted Display (HMD) acts as an intermediate to teleport her from the real world to the virtual world. Various motion-controllers provide the super-power to to move around and perform realistic interactions and activities in the virtual environment.”

Now let me map each component in the definition of VR to my developmental setup:

  • Immersive or 360 degree environment ====> UE4 environment
  • Head-Mounted Display (HMD) ====> Oculus Rift
  • Motion-sensors ====> 5DT Data Gloves 14 Ultra
  • Dynamic or Realistic Simulation ====> grabbing objects, picking objects, throwing objects, etc.

The video below will certainly increase your curiosity about Virtual Reality.

Next coming up for you is Virtual World.

What is Virtual World?

Virtual World is nothing but a complete 360 degrees immersive environment developed using game engines such as Unity 3D, Unreal Engine 4, etc. A Virtual World serves as a complete package of both static and dynamic objects with various physical properties and behaviours made possible by detailed physics simulation.

To develop such 360 degrees environments we will need a platform which can serve all our requirements. Nowadays there are various gaming engines available for development of such immersive environments of which the most popular are Unreal Engine 4 and Unity 3D. I prefer to work on Unreal Engine 4 because of its diversity and versatility — all of which is available for free.

Let me map my components to virtual world :

  • Static objects ====> 3D boxes, 3D balls, 3D Hands, a platform.
  • 3d boxes, 3D balls ====> These are static objects which will be used for performing actions like grabbing, throwing, etc with the help of 5DT Data Gloves 14 Ultra.
  • A platform ====>The surface where the player or user can stand to carry out the operations.

What is Unreal Engine 4?

Source
Unreal Engine 4 is a complete package with professional tools and latest technologies which are used to create high-definition games for different platforms such as Android, Windows, etc. The rendering architecture of Unreal Engine 4 allows developers to create or develop surprisingly highly interactive environments from high-level systems to low-level systems.

Unreal Engine is a powerful platform with a revolutionary new workflow and features a diverse set of tools that developers can use to quickly iterate ideas and see results immediately. Full accessibility of the C ++ source code is the icing on the cake for developers.

If you want to choose UE4 as your development platform then the following series of videos should be your first footsteps.

Tutorial video for plugin development

What is 5DT Data Gloves 14 Ultra?

Source
5DT Data Gloves 14 Ultra is a hand data motion capturing solution for animation and virtual reality. They are designed to satisfy the stringent requirements of modern Motion Capture and Animation Professionals. It offers comfort, ease of use, a small form factor and multiple application drivers. The high data quality, low cross-correlation and high data rate make it ideal for realistic real-time animation.
Source
  • 5DT Data Gloves 14 Ultra are devices which are capable of replicating our hands and fingers movements and their gestures with available software interface. The Data Gloves are used in form of input device and serves the role of motion controllers in virtual scenario.
  • Data gloves have the presence of 14 flex sensor at different position of our hand which are capable of performing reenactment real hand movements and different gestures easily.
Features of this glove
  1. The glove is comfortable with an improved design for differently-sized hands.
  2. The accurate and sensitive sensors give a clean signal, minimizing the need for additional filtering.
  3. Diagnostic software is included with the capability to record hand data.
  4. The package optionally includes a plugin for Autodesk MotionBuilder.
  5. The gloves functions and data are accessible via the 5DT Data Glove SDK.
  6. The glove data stream from the optional serial interface kit is an RS-232 data stream with an open protocol allowing for various platform implementations.
Source
Working Principle of 5DT Data Gloves 14 Ultra
  • The major components responsible for working of data gloves is Flex Sensors present in gloves at different position.
  • With the fingers movements there is movements in flex sensors respectively which is responsible for the collection of data and further for the reenactment.
Source
What is a Flex sensor?

Flex sensor is none other than the strips which are variable resistors present in the data gloves. The input that we receive from the gloves depends upon the bending angle of the flex sensor and so this reading may vary at different instance of the hand movements.

Source
Sensor Mappings of 5DT Data Gloves 14 Ultra
Source
Hand Gestures of 5DT Data Gloves 14 Ultra
Source
Have a look at the video representing hand movements with Data Gloves
Demo Video on how reenactment of real hand movements on virtual hand
Plugins Support from the 5DT Team:

How I managed to build and deliver the plugin

Now I will describe my journey of the successful development of an UE4 plugin for 5DT Data Gloves 14 Ultra.

Let me start with how I approached the problem:

Stage 1

First of all, my target was to test whether the sensors present in the gloves are responding properly. For this, we will plug the gloves into a PC and fire up Glove Manager, a tool which packed with the Gloves in a CD.

Source

Resources available at this stage:

  • Glove Manager
  • 5DT Data Gloves 14 Ultra

Check out how the Glove Manager app works in the video below:

Courtesy : SAVR Lab, Indian Institute of Technology, Kharagpur

Conclusion after Stage 1: At this point it was clear that the gloves were working properly and we were able to get the proper sensor feed backs. It was the most important step that needs to be verified in the beginning.

Stage 2

I started looking for the available plugins for 5DT Data Gloves 14 Ultra. The purpose of searching for other plugins was to check how the gloves should be responding and what pipelines were being used to set up interaction with virtual objects.

Resources found at this stage:

  • Plugin for latest Autodesk MotionBuilder.
  • Plugin for Vizard 5.
  • Software requirements for Autodesk MotionBuilder, Vizard 5
  • Hardware requirements for 5DT Data Gloves 14 Ultra

Resources unavailable: C++ SDK for 5DT Data Gloves 14 Ultra

Now, let me give you a brief how the Data Gloves responded in Autodesk Motionbuilder and Vizard 5.

My developments and testings :

  1. Autodesk Motionbuilder

I tried testing the gloves on Autodesk Motion builder following all the instructions in the Data Gloves manual. It worked fine in Autodesk Motion builder as mentioned in the documentation.

Dis-advantage : The problem at this stage was that although the gloves were showing proper hand movements in play mode but they did not serve my objective of re-enactment of gestures on virtual objects. We needed a gaming platform.

2. Vizard 5

Further, I came to know about plugins available for Vizard 5. So, I went on to try 5DT Data Gloves 14 Ultra on Vizard to test how hand movements and simulations caused due to those movements look like. Vizard 5 has a python based development environment. So, it was pretty easy for me at that point to quickly test the gloves. I tested and here are some snapshots and videos of the same:

Courtesy : SAVR Lab, Indian Institute of Technology, Kharagpur
Courtesy : SAVR Lab, Indian Institute of Technology, Kharagpur
Courtesy : SAVR Lab, Indian Institute of Technology, Kharagpur

So, these were my initial steps to test whether the real hand movements recorded by the Gloves could be reenacted on virtual 3D hand models. Next, I tried to check whether it was able to perform various physics simulations in a 3D environment. Here is a video which helped me understand more clearly how the Gloves would actually work. Check this out:

Courtesy : SAVR Lab, Indian Institute of Technology, Kharagpur

Conclusion after Stage 2 : At this point I was clear that the Gloves were properly calibrated and each and every sensor was responding to the real hand movements. It was confirmed that if we manage to map the sensors correctly to a 3D hand model accurate reenactment would follow.

P.S : At this stage, the C++ SDK was not available to me, to start my development in Unreal Engine 4. I had a conversation with the 5DT Team over email and they were quick to release the SDK. I deeply appreciate their fast response and support. I was all set to start my development.

Stage 3

It was time to build a PoC (Proof of Concept):

Resources available at this stage :

  • C++ SDK for 5DT Data Gloves 14 Ultra
  • Software platforms: Visual Studio 2015, Unreal Engine 4
  • Hardware platforms : 5DT Data Gloves 14 Ultra, Oculus Rift, Oculus touch controllers
  • 3D Assests : 3D hand model, few 3D objects like ball or boxes.
P.S. : While writing scripts for the Gloves, the two things should be kept in mind (a) Whether we want to gloves work with the help of hand gestures.(b) Secondly, or we want to make gloves work with the help of real hand movements.
Windows Emulator for 5DT Data Gloves 14 Ultra(Simulation based on Gestures)
  • Now let me introduce you to one of the procedure to make interaction with gloves possible for windows platform. If we desire to make interaction based on gloves gesture the best possible way is to go for VRPN (Virtual Reality Peripheral Network).
  • To make the gesture based interaction possible, VPRN there is a best possible way to make it happen. For this we need to create a Windows Emulator for 5DT Data Gloves 14 Ultra. Emulator can be created with the available VPRN solution for Data Gloves.
  • In this case Data Gloves solution will be universal, by the term universal I mean it will interact across all the windows applications with help of gestures.
  • The procedure which I followed here in development is that I mapped each gestures of the gloves to the individual alphabet key present on keyboard. The functionality of gloves is like whenever a user will perform one gesture that is mapped to the corresponding keyword on keyboard. So, here gestures will perform as an input similar to the keyboard inputs.
  • In case of Unreal Engine while development of any environment every simulation occurs with the help of respective keyboard button. So, whenever a user performs some gesture in run-time, the corresponding simulation attached to that key will occur in the environment.
Unreal Engine 4 plugin for 5DT Data Gloves 14 Ultra

For development of Ue4 plugin as a first step, I tried the basic VR template in Unreal Engine 4.13 to check the accuracy of the hand movements in case of Oculus Rift. Here is a sample video demonstrating on how actual reenactment work in Ue4 with Oculus Rift:

Demo implemented on Unreal Engine 4 for Proof of Concept

From the video I was pretty clear by now how Data Gloves might work in Unreal Engine. So, I came to the conclusion that we need to map all the sensors to their respective positions on a 3D hand model. So, I started my development.

Basic working structure to the development

Here are the steps that I followed:

  1. Open up an Unreal C++ project in the unreal editor and select a basic project package and open up the project.
  2. After launching the project go to Edit->Plugins and select the Blank Plugin to enable it. This will prompt a message at the right bottom corner to restart the engine.
The playlist that will guide you to the basic of the development
  1. After the engine restarts, just copy the Blank Plugin folder inside Developers folder within the Plugins directory to a New Folder within the same Plugins directory and change the name to 5DTDataGloves14Ultra. Rename the directories respectively, files and code contents to match our new Plugin name.
  2. Add the appropriate files provided in the C++ SDK of 5DT Data Gloves in the appropriate directories created above (files from SDK includes the .h, .cpp, .dll files {most important take care of the *86 and *64 based files}) and start writing scripts for the two gloves respectively.

The screenshot below will help you in adding the files properly in folders.

Screenshot taken from Unreal Engine whitepaper titled “Custom input device plugin guide-Epic Wiki”

3. That’s it! Compile the entire project and the data gloves ready to be tested.

Check out the following link for more details:

Packaging the 5DT Data Gloves 14 Ultra Plugin

Finally we have to create the package of our gloves plugin to use it further on any device. For this we need to follow this steps : Edit — > Plugins menu and select the developed plugin — >select the Package option. The editor will automatically build the plugin for all the supported platforms and create a folder. To use our plugin in another project, copy the Plugins folder (it is located inside [our_Export_Folder] — >HostProject) inside your desired project. Please note that the target project has to be a C++ project.

Snapshot of the plugin window
Procedure for using the plugin in Unreal Engine
  1. After plugin package is successfully build, just restart the unreal engine editor. We just have to follow the same procedure as above i.e., go to Edit — > Plugins menu and select the developer plugin — > 5DT Data Gloves 14 Ultra.
  2. Next, we will get an option at the right bottom corner of the plugin window that will instruct us to restart the engine after saving our entire project.
  3. After the engine restarts, Voila! our plugin becomes entirely active for use in the environment :)

That’s all about it. If you made it till the end, thank you and hope you found it useful… Please feel free to use the comments section to suggest modifications and/or additions. Finally I would like to convey my heartfelt thanks to Prof. Pabitra Mitra and Prof. Jhareswar Maiti for providing me an opportunity to taste this wonderful domain of VR. I had an amazing experience while being a part of this project at IIT Kharagpur which was under the Uchhatar Avishkar Yojana (UAY) scheme, jointly funded by Tata Steel and Ministry of Human Resource Development, Govt. of India. Last but not least I would love to convey my sincere thanks to Anirban Santara for his constant guidance and support throughout my tenure at IIT and my labmates from SAVR lab, IIT Kharagpur for their co-operation in making this work a success.