Making Faces — Computer Vision Research at ObEN
ObEN is actively looking for visionary talent to join our team and help develop the next generation of computer vision technology. In this blog, we offer an intimate look at the technology that powers our Personal AI and some of the projects our CV team is working on. Want to know what it’s like working as a researcher at ObEN? Read our blog here. If you want be a part of our world-class team of scientists, researchers, and engineers visit the ObEN career page to learn more about our current openings.
If I asked you to describe the face of someone you loved, what would you say? Would you start with the shape of their face: heart, oval, square? Would you mention the contours of their profile, the sharpness of their bone structure, the color of their eyes, the texture of their skin? How many characteristics would you go through before you are satisfied with the accuracy of your description?
At ObEN, the task of making faces falls to the researchers and engineers of our Computer Vision team, who spend every day figuring out how to accurately describe a face. Their work has enabled us to create machine learning based systems that allow our models to take a single selfie and convert it into a 3D rendering of a person’s face, capable of emoting, speaking, and bringing our Personal AI (PAI) avatars to life.
Generating your likeness in avatar form is a tall order. While most of us can think of a handful of points to help map out a face, our CV team looks at dozens of landmarks on each 2D image and analyzes how each of these landmarks shift and change between 2D and 3D, morphing a 3D model.
It starts with a face database that we’ve built in-house, collecting vast amounts of data that help us determine these facial landmarks and train an AI algorithm to recognize them as well. By comparing 2D images to their 3D counterparts, we analyze how these markers deform and shift between 2D and 3D. The AI is then trained to automatically deform (or reform) a generic 3D head model (a mesh wireframe) so that it matches the face depicted in a 2D image, with landmark points annotated and checked for accuracy. In many ways, our work involves teaching a computer how to describe a person’s face — recognizing the variation between people, the shapes and contours of their face, and analyzing those characteristics in a statistical way so that the computer can map out these variations to create the face of an individual.
Lifelike is good, but too lifelike can be too much. All of ObEN’s teams work closely together in a collaborative ecosystem, which is why our researchers learned early on from our product team that users expect their avatars to look like an enhanced, “prettified” version of themselves, rather than an exact replica. Which is why they could tackle training the AI to generate avatars that presented a good balance between mirroring and improving the person’s look, bridging the gap between art and technology.
To quickly complete this process, the AI must be taught to recognize how variations of these landmarks (in other words the texture of your individual face) can be merged with the default wireframe to best simulate your visual appearance. Like artists in traditional mediums, some attributes are more important than others — and subsequently harder to capture. Special care is taken for areas around the eyes and mouth — the features that most betray our emotions — to ensure our PAI adopt a lifelike quality. In addition, our CV team is also exploring automating how the PAI’s face animates when speaking. Our research into visual speech involves training the AI to recognize how words are spoken by various people, the animation of the face and especially the movement of the mouth. With enough training, the AI can then take any text or audio input and automatically animate how a user’s PAI would say those words based on their facial characteristics. Just imagine having your PAI reading bedtime stories to your children when you are traveling, just as you would — our visual speech technology is making that possible.
For most users, this is the technology they are using when they create their own Personal AI from one of our apps. The CV technology lets users create a PAI that looks like them with a single selfie. Combined with our Speech and NLP technology, these digital doppelgangers can talk and behave like them as well. Eventually, that PAI can serve as a digital representation of you. Anyone interacting with your PAI can feel the personalized experience of looking at you and interacting with your likeness. And don’t fret about someone else trying to create digital copies of you or take control of your PAI — all PAIs are authenticated and secured on the Project PAI blockchain to ensure you are just as in control in the digital space as you are in real life.
In addition to our consumer technology, our team also creates studio quality avatars for our strategic partners and celebrity clients. Through a joint venture with SM Entertainment — South Korea’s largest entertainment agency — ObEN established AI Stars, the world’s first celebrity AI agency. This means we are creating the PAIs for some of the top celebrities in the world, so that they can create new experiences and connect with fans all over the world. The underlying technology is similar to our consumer technology, with our teams harvesting more data through a full 3D face scan of the celebrities we work with. With more data, the model has more to work with, resulting in a higher quality avatar.
But, for ObEN researchers, a pretty face isn’t enough. Our ongoing work not only involves constantly improving our AI model, but also creating customized facial rigs for more expressive avatars. We are now looking at data sets from a wide range of expressions, analyzing the corresponding markers for these different facial expressions and even teaching the model to predict the facial shape of different expressions. Soon, your PAI won’t just look like you — it will cry like you, smile like you, and laugh like you. A PAI worthy of your face.
Join our Community
Our newsletter subscribers get exclusive access to beta applications and news updates. Subscribe here. Follow our journey on Twitter.
