iPhone X’s 3D Depth Camera and the Unfolded Scenery of the Future
Apple has always been on the frontline of innovation. A small personal computer — a.k.a. ‘a smartphone’— is now in each of our hands just as Steve Jobs dreamed of. A smartphone changed our lives enabling so many possibilities and we now cannot imagine a life without this small device. And the announcement from Apple’s keynote on September 12, 2017 was an awaited moment for iPhone fans with the promise of a brand new iPhone celebrating iPhone’s 10th anniversary.
And, as we expected, Apple did not let us down.
Missing ‘Home’ button, Animoji, and FaceID seems not that impressive. But beneath the functions, there is 3D camera integration and AI algorithm including facial tracking technology. Ultimately, this foundation stone is the implication of AR, Virtual Assistants, and Gesture Interface beyond the smartphone in a flat square shape.
But leaving the iPhone X’s general butterfly effect for a different day, let’s talk about facial tracking technology and its impact first.
What thrilled us the most was that the new iPhone X integrated 3D depth camera and applied facial tracking technology. As you know, BinaryVR develops facial tracking technology to bring humanity into our virtual experience. We are also the company possessing 3D facial motion capture technology identical to the iPhone X’s. The new features — Animoji and face ID — will be the starting point of how our daily lives will be changed.
Imagine the future and the unfolded scenery with the facial tracking technology. How would Apple accelerate the future of our lives again as the smartphone assimilated into modern society? Beyond the smartphone, the 3D depth camera will be inserted in smart home devices such as smart speakers and automobiles facing ourselves in every moments.
What’s in iPhone X?
First thing first, where and how did iPhone X apply the facial tracking technology? We will show you what kind of technology is used in iPhone X with their Animoji and face ID features.
Animoji — Real-time Facial Motion Capture Technology
Apple presented the Animoji feature tracked upon iPhone X’s 3D camera sensor. With Animoji, we can now breathe our own personality into our favorite characters. In other words, avatar experiences in mobile will be a powerful tool for storytelling and content creation. This might enable ordinary users to easily create short, light animation film putting aside any specialized skills. Fortunately, iPhone already has a record of empowering users as talented photographers!
This is the technical process to enable Animoji.
Detect a user’s face
Calibrate the user’s face
Track facial features and movements
Track facial expression values
Remap the values into a CG character
The whole process is called 3D facial motion capture technology. As BinaryVR already developed the algorithm optimized for commercial uses, we will explain the detailed process in the next article.
Face ID — Facial Identification Technology
Face ID is one of biometric identification that will ultimately replace Touch ID. Thinking underneath the surface, the power of Face ID is in even more improving mobility. With a smart watch, a smart speaker, or inside of your car, you can fully utilize your smart devices without your hands.
Here is the technical process of Face ID.
As I look at the phone,
The 3D sensor detects the face
And recognizes the eye gaze to the phone
The algorithm identifies the face and the owner’s facial characteristics
the screen is unlocked.
Face detection, eye gaze tracking, and facial identification algorithms are all applied to enable Face ID.
Apple’s Animoji and Face ID features are the result of combining 3D depth camera sensor and facial tracking technology. Apple just opened up the door of commercial level facial tracking technology to our daily lives.
The Unfolded Scenery of the Future
The first move of Apple would accelerate other smartphone manufacturer companies to integrate 3D cameras and facial tracking technology. As the time goes by, the price of 3D cameras is becoming cheaper and now it is down to the commercial use price. Starting from the smart speaker and other smart home devices, the 3D camera will be soon integrated in all IoT products and AI robots. Of course, we cannot count out automobiles — the hottest trend in 2017 thanks to Tesla.
Along with this context, Apple’s HomePod means something more than upgraded Siri 2.0. It takes only a bit of consideration to imagine the next version HomePod with a 3D sensor and facial tracking technology. When the imagination becomes true, how would our lives be shaped? Here’s an example.
Smart Home Devices that Recognizes Your Family
As you come home, the smart speaker detects and identifies your face with its IR sensor and turns the light on. And you look at the speaker and say, “Please turn on the TV.” Your eye gaze becomes the trigger and the speaker recognizes your voice over other sound noises. Landmark tracking allows the speaker to synchronize your voice and your mouth movement to ensure your request.
A few minutes later, your two kids and husband come home. As soon as they come home, the speaker recognizes your kids and changes the TV channel into a non-violent one as you set as children safe mode. The speaker captures the expression of your daughter and analyzes her emotion as happy and you get the notice of her status from the speaker: ‘Bella’s status — Happy.’ While Bell’s favorite show is on the TV, Bella also becomes one of the character with the facial motion capture technology actively participate in the storytelling!
Self Driving Car with Enhanced Safety and Entertainment
The next morning, you are about to go to work, and hop on the car with your kids. The car’s 3D camera sensor recognizes you and your kids by the facial identification and unlock the door immediately. As the car recognizes your kids, it asks you if the destination is your kids’ school. After your kids get off, as an enthusiastic driver, you take the wheel and start driving changing the automatic driving mode into manual. While driving, on the radio channel, a related commercial comes out, upon the attributes of family members.
As time goes by, you feel tired and you start to doze off while driving. The car senses the frequency and timing of your eyes blinking to diagnose fatigue. As the car captures your facial emotion as ‘sleepy,’ the manual mode becomes automatic mode and recommends to sleep. You nod your head meaning yes, and your chair adjusts all the way down automatically.
Facial Tracking Technology in Our Daily Lives
Through Apple’s Animoji and Face ID, we now can imagine the future with facial tracking technology in our daily lives. The acceleration will be faster and more powerful than ever. We are thrilled to see what would be possible with Apple’s new feature. As a first step, we will support innovative mobile app developers to create AR apps utilizing facial motion capture technology.
Moreover, there are big tech companies developing Smart Home devices and AI robots. Automobile industry is on the hype of AI system integration in cars including self driving function. And many of ambitious IoT companies are reaching out to us to integrate facial tracking SDK.
In the near future, BinaryVR’s facial tracking technology will be utilized in our daily lives as we imagined.
Follow us and check out our blog to be updated on BinaryVR’s vision and technology! BinaryVR brings humanity into the virtual world from the very front line of cutting-edge technology.
