Personalized Hand Models for VR

For a truly immersive experience

Dávid Komorowicz
Sep 20, 2018 · 6 min read
Hands in The Gallery — Episode 1: Call of the Starseed

I want to transform the character’s body to resemble the player’s own physical characteristics, making games a truly personal experience.

Recent advancements in Artificial Intelligence made it possible to take personalization to the next level, whether generating voice on the fly (TTS) or modeling the player’s face from a single image.

Online Demo

Take a look at my interactive demo here. You can change the shape of the hand by adjusting the sliders on the right.

Deformable Hand Demo

The Model

As a starting point, I’m using the Mano model. It is a parametric hand model published by the Perceiving Systems Department in Tübingen.
I’ll get into more details in the following article. If you are interested, make sure you’ve clicked the Follow button next to my name.

Skeleton with Blend Weights

Executing Python code in Maya

Leap Motion

Hand model controlled by Leap Motion
Leap Motion joint orientations (Source)

Goal

The end result of this project will be a mobile app that can transform hand photos into a textured 3D mesh consisting of 10 shape parameters and the different texture files (albedo, normal, etc). These will be extracted with Convolutional Neural Networks (CNN), which is the state of the art method for image processing tasks. These will then be transferred to a supported game of choice. The model can be animated with conventional methods thanks to the MANO model we use as a base.

Mobile app concept

What’s next?

Currently, I’m finishing up cleaning the training data and starting to train a neural network. In the next post, I’ll write about the results of the first prototype which will estimate the shape parameters and what I’ve learned along the way. In the meantime follow me to get notified when it comes out.


Kitchen Budapest Stories

Collection of articles written by members and peers of Kitchen Budapest, the Open Innovation Lab of T-Systems Hungary. Articles about technology, innovation, digital literacy, DIY, research and education.

Thanks to Nagyfi Richárd.

Dávid Komorowicz

Written by

Machine Learning Engineer interested in Deep Learning and Virtual Reality. dawars.me

Kitchen Budapest Stories

Collection of articles written by members and peers of Kitchen Budapest, the Open Innovation Lab of T-Systems Hungary. Articles about technology, innovation, digital literacy, DIY, research and education.