Body Tracking and AR?
Let’s talk about “Body Tracking technology” today.
Snap Inc. provided a new line of Snapchat Lenses that combine 3D motion capture and multiple neural networks to create over 10 augmented reality costumes powered by 3D full-body tracking.
The company also recently announced Snapchat integration with Apple’s new LiDAR Scanner powered by the iPhone 12, allowing for more realistic AR experiences by generating a “mesh scale” of your environment.
This new technology announced by Snapchat will allow users to virtually wear various costumes. In the past months Snapchat has implemented various 2D tracking technologies and will no be adding a 3D tracking technology that will analyze over 18 joints of the human body to enable more realistic tracking in 3D.
Using this technology users will not only be able to dress themselves up in virtual costumes but also add virtual back dancer to create videos like pop stars. The lense will enable both android and iOS users to wear various costumes ranging from the iconic dancing hot dog, a Jacko lantern, or a skeleton. Snapchat has been a leading service and has led the video format texting and social media.
Body Tracking?
Body tracking can be rephrased to human detection. Open Pose is the one of the popular detection framework. It seems that it can detect 135 key-points!
Also, some tech company and OSS team is tackling this domain. You can find it in GitHub. https://github.com/search?q=body+tracking
For example, Facebook makes VideoPose3D and detectron2.
What is body tracking for AR?
For output, you can get idea like Snapchat above. Also, for input, this technology make accurate detection of person before record..
However, I think it is not Core Essence of body tracking for AR and MR.
I need time to think this..
Because,,
I will show one sample reason, below.
For VR?
Someone made demo for making Virtual character dance video with using body tracking technology.
You can use any framework for these work.
Importantly, you don’t need LiDAR scanner for it.
It means that snapchat can use these technology with using normal camera.
I feel that snapchat could not be able to find core essence of this body tracking technology.
so, It seems that it is difficult to find core essence of this body tracking technology in normal situations or normal environments.
Maybe you can find it in extreme environment..
