An Era of Digital Humans
Digital humans, which can be brought to life with AI, will be important for a number of industries
By Simon Yuen, Director, Graphics and AI
We have been pursuing the creation of digital humans for years.
Traditionally, digital humans have been widely used in media and entertainment, from video game characters to CGI characters in movies. But the process to create a digital human is extremely labor intensive and manual. It requires hundreds of domain experts such as artists, programmers, and technical artists. Plus, humans understand when an artificial human is fake. We’re extremely sensitive to the uncanny valley effect, so we will know when something is off.
NVIDIA is researching tools and developing ways to accelerate and simplify digital human creation — and we believe AI and simulation is the key to doing this.
What is a Digital Human?
At the core, a digital human is a digital form of ourselves in the virtual world. From the early days of 3D games, Virtua Fighter was one of the first to show how 3D characters can fight together. And today, players can experience the journeys of memorable characters from popular games like God of War and The Last of Us. Or maybe you recognize digital humans in popular movies, like the infamous villain Thanos from Avengers: Endgame, or Brad Pitt as the older version of himself in Benjamin Button.
There are also new cases emerging in entertainment through storytelling with digital avatars. Companies like Fable AI and Baobab are creating interactive and virtual stories that involve digital characters. But how do we define what a digital human is? What metric can we use to describe different types of digital humans?
There are generally three scales or ranges, on which we measure digital humans: Realism vs. Stylism, Real-time vs Offline, and AI vs. Human Driven.
Bringing Digital Humans to Life
Like previously mentioned, the process to create a digital human can be challenging. There are three main components to making a digital human, and each of these requires a different combination of art and technologies: Generation, Animation, and Intelligence.
To generate digital humans, teams must first make 3D models, textures, shaders, skeleton rig, and deformation of the skin so it follows the skeleton.
For animation and movement, artists must look at the physical elements of the digital human, from the body and face to hair and clothing. Typically it is a combination of deformation and simulation to achieve the right motion for these parts. In terms of realistic performance, up until now, there are mainly 2 ways to achieve this; animate by hand or use various performance capture techniques to get your motion data. Often it is the combination of the two together. Within the last few years, using artificial intelligence (AI) to generate or synthesize animation is starting to appear more. Their roles are smaller now and typically handle specific types of performance, but that is changing rapidly. All in all, the bottom line of any of these approaches is to create context based behavior so that the digital human can act or move in a believable way. Being able to show emotion and behavior like a real human is still very difficult. But new technologies in AI and simulation are helping to make this easier.
And lastly, artists must bring intelligence to digital humans, and they can do this through bidirectional interaction. Through Human Language Processing and Natural Speech technologies (Riva, Ensemble AI, Replica), a digital human can be able to have conversations with real humans. They will also have vision within the virtual world as well as the real world. They can recognize objects and surroundings in their environment so they can navigate within it. They will also be able to see the users that are talking to them so they can look and respond accordingly.
The Importance of Digital Humans
Digital humans may be in media and entertainment, but the need for digital humans is growing, and it’s spanning across industries. Today, we are already seeing all the benefits and potential use cases for digital humans.
To start, the use of AI digital assistants has great potential in industries like healthcare and retail. For medical professionals, digital assistants can help improve training and procedures. Doctors can operate in a realistic simulation, and they can conduct the simulation hundreds of times to ensure they get the best results before performing the surgery in real life. And in retail, AI digital assistance can enhance customer services by providing a more personalized experience.
But for this to work, AI digital assistants need great comprehension of verbal communication — this is key to helping people better interact and converse with digital assistants, so they can accomplish the tasks they need.
For other companies that work in industries like architecture and manufacturing, digital twins are helping teams simulate workers and people in large environments, from factories to cities to buildings. With the help of digital humans, companies can assess risks and predict environments with accurate simulations, helping them ensure the physical building will be optimally designed.
The intelligence requirement in digital humans will be different from AI digital assistants — when putting a digital human in a virtual environment, they’ll need to know how to navigate and behave like a person, whether it’s a factory worker or a tourist walking through a skyscraper.
Lastly, digital humans will help improve the creation of synthetic data generation. For AI, data and training neural networks are the essence — companies like Synthesis AI, Microsoft, DataGen, Epic Games, and Reallusion are already working on capturing and synthesizing 3D digital human data to train AI models. But we will continuously need more data, especially for the future. Synthetic data generation is key for AI to grow, and synthetic data generation of digital humans will be crucial to expanding that.
Every Voice Will Have a Face
What’s on the horizon for digital humans? People will have a very creative usage of an adopted digital human world, and as we move towards experiences in virtual worlds, this will become more prominent.
Digital humans are essential for virtual world experiences — in fact, everyone will one day have their own digital version of themselves — whether an accurate or stylized avatar.
With NVIDIA Omniverse, we want to create the framework where many types of digital humans can coexist. Using Pixar’s Universal Scene Description (USD) is the standard format across all 3D industries, everyone can exchange data and talk together. Omniverse is helping drive these efforts towards USD, and this is key to enabling different applications and technologies to bridge and collaborate to create digital humans.
And for large worlds with tons of digital people, we have to be able to scale up. Simulating large worlds that can interconnect, and having large amounts of digital humans that can interact together in real time, is a very challenging computation problem. But Omniverse is addressing this challenge — the platform can power large worlds and simulations. This will be crucial for the future of physical and virtual worlds, where you can have massive amounts and varieties of digital humans that can participate and interact together.
Over time, the connection between real humans and digital humans will grow. It will go beyond watching a puppet on the computer. Eventually, the computer will read and interact with us, just like what we do in real life.
We’ll be able to talk with digital humans, and even order goods like merchandise, food, prescriptions, and other things through a digital person, who will then deliver tangible real-world objects to you. Communication and interaction will become a two-way street, and this provides a new element of freedom and reinvention.
Join us at NVIDIA GTC to see where the future of digital humans is heading. Register for free, and check out our session on Digital Human Technologies, our Expert Breakout, and the NVIDIA Omniverse User Group to take an in-depth look.
And don’t miss the special keynote, presented by our CEO Jensen Huang on Nov. 9, to hear all the latest technologies in AI and graphics.
See the Omniverse Developer Resource Center for additional resources, view the latest tutorials on Omniverse, check out the forums for support or join the Omniverse Discord server to chat with the community.
Simon Yuen is Director of Graphics and AI at NVIDIA where he leads the Digital Human efforts with the goal of developing new character technology and deep learning-based solutions that allow new and more efficient ways of creating high-quality digital characters. Prior to NVIDIA, Simon spent more than 21 years in the visual effects industry, straddling both the art and technology sides of the problem at many studios including Method Studios, Digital Domain, Sony Pictures Imageworks, Dreamworks, Blizzard Entertainment and among others building teams and technologies that push the envelope of photorealistic digital character creation.
