What is Animation Retargeting?
Learn the basics of animation retargeting, how it works, and common challenges
By Lina Halper, Principal Animation Engineer, NVIDIA
Have you ever mimicked somebody else’s behavior? Have you tried to act like somebody else?
It is quite easy for us to mimic somebody else in the real world. What about in the 3D world? You see walking motion in one character, but how hard is it to use the same motion for other characters? Sounds like it should be easy like a copy paste problem, right?
Animation retargeting is a problem where you want to play the same animation on different characters. This seems reasonable, especially if you look between humanoids as they look very alike. You can picture an animator easily applying the same motion on different characters.
This is one of my favorite topics. Anyone who has worked with me over the past decade knows how much I love talking about this, and it is still a challenge.
Why is that? It is complicated to explain. All joints, hierarchy, and skinning — it’s all a very difficult topic and people get lost quickly in the discussion. With digital humans, synthetic data generation, and the metaverse, we have a huge crowd of people who are new to 3D.
And now, the creation of 3D avatars is a topic for everybody, not just the traditional video game and cinematic crowd. But it’s a long rigorous process to explain what types of problems there are with motion transfer. Also, physics! Since we’re so familiar with it in our daily lives, we know exactly how it should work in our world — but not in a virtual 3D world. We have to implement all that information back to it and it is not trivial work. How can we make this easier?
What Are the Challenges?
At the core of animation, we control motions with joints (ignoring blendshapes). Joints are points in the space to describe transforms like this.
Joints make up hierarchy in a 3D space with a parent-child relationship. That relationship is shown with lines and this forms more like a character that you’re familiar with.
With this, now we can move the joint like in this gif. But now, how do we move the mesh with it? The above gif is probably not what you have in your mind when you move the joint.
The binding of joints and vertices is called skinning. The skinning is to deform vertices with the joints, and it can cause many issues depending on how you want the character to deform, so there are many extra joints added to fix those issues.
In the 3D world, vertices don’t know about our expectations until we implement all of them. They don’t know they should not cross each other until we make them. So odd twists, loss of volumes, or gain of volumes, all become issues that we don’t see in our daily lives.
With all these, when you have very different characters with different types of joints, motion transfer becomes that much more complicated.
But when we simply look outside, we can easily picture in our head how the animation should look like on different characters.
Trade-Off of Simple Expression For A Complex Problem
How can we narrow this gap? How else can we describe a skeleton?
When you train AI using motions, you want the minimal information, so you want your skeleton to be as simple as possible.
This is great until you want to retarget this motion to sophisticated characters. You need an Inverse Kinematic retargeter than Forward Kinematic retargeter, since the source information is not rich enough.
Many image-based AIs that generate motion, such as Pose Estimation, show different representations of humans. They’re usually made of lines in 2D. This makes it a bit easier for people to understand, since it’s also 2D and we know limbs.
Is this a better way to describe the character? But how will we describe the complexity of twist and skinning?
Or maybe we can use lesser descriptions, like looking at the hand and foot locations, and solve it by using a combination of traditional retargeting and IKs to reconstruct motion. This solves the retargeting problem at the core, since you’re detaching hierarchy as a separate problem. But could it be used to create most of the motions we’d like? For example, can we express all the walking motions like this by using this approach?
The gap between how we see humans to how it’s implemented at the core is huge.
We want users’ understanding to be more straightforward and hide the complexity underneath. The surface should be easy for users to comprehend while the underneath should allow the complexity to implement all the details of the expressions we’d like to see.
How can we represent characters in a way that’s easy to understand, while allowing implementation to express the complication in it?
One Click Animation Retargeting
NVIDIA Omniverse is seeing many users who are just beginning to work in 3D worlds, and it is our goal to create tools to make this process easier for them. We call it “One Click” — we want it to be as easy as just one click to achieve something. We’d like to transfer the animations to the target while retaining the motion and intentions as much as possible with One Click.
For us to create One Click, we have to see the traditional problems from different angles. If you want to see how we tackle this problem in the Omniverse, join me at GTC on Thursday, March 24, at 8:00 AM PST. I’m going to talk about how we see character motions in the talk and how we achieve “One Click Animation Retargeting”.
We have a very exciting release coming for animation in Omniverse — it’s not only for enhancing runtime animation, but also for generating motions using AIs, as well as to retarget those motions easily.
For those attending GDC, join me to see how game developers can use this to enhance current pipelines or workflows, and get a sneak peek into upcoming Omniverse animation tools.
I’m very excited to finally talk about what I’ve been working on. I will dive deeper into how this “One Click” system works, so come join the talk and learn more about this delightful, fun topic.
Meet the Author
Lina Halper, Principal Animation Engineer, NVIDIA
Lina Halper has 20 years of game engineer experience from Battlefield, Gears of War, Paragon, and Fortnite. During her 12 years at Epic Games, she focused on motion and animations of characters. She led the UnrealEngine Animation System & Tools effort, developed the Facial Animation System(Pose Blending) and the ControlRig. She was the tech lead for MetaHuman Creator and was also involved in various special projects such as HellBlade Senua Live GDC, Siren, and LOTR Smaug VR. Currently, she is a Principal Animation Engineer at NVIDIA working on Omniverse, a real-time simulation and collaboration platform for 3D production pipelines. She loves animations as a medium of storytelling through motions and expressions and has a huge passion to develop tools to tell the stories by anybody as easily as possible.
