Face Tracking in Blender Using A Webcam, OSC, and AddRoutes

Introduction

This is the transcript version from the original youtube video recording. You can watch here or by clicking on the image below if you prefer to follow along.

Youtube Video

All right we’re going to start this out by just giving you a little preview of where we’re hopefully going to end up at the end of this tutorial. By the end you should be able to have a model that is completely rigged with 52 blend shapes using face it and you should be able to control it just using your webcam for facial expressions as well as rotation and translation.

Getting Setup (0:27)

To give you a little preview of what it’ll look like at the end, let’s just take the model that we’ve already rigged up. Just go to FACEIT. Make sure Face Cap is all set up. We’re just going to open Hallway.

• Go to Customize.
• Open up OSC Streaming and turn it ON.

Rain, she disappeared, hit dot to recenter. There we go. As I move my head, she moves her head and she’s tracking all my movements inside Blender. Barely real-time. That’s because she’s got kind of an aggressive texture bake, but we will cover that later. We’re going to stop this and we’re going to start at the beginning. Don’t save. So yeah, in order to get started, the first thing that you’re going to need is to use Blender 3.0. In about a month the plug-in AddRoutes is going to be updated so it can run on 3.1 or 3.2, but today you’re going to want to use 3.0.

• Go to Hallway.
• Go to Downloads.
• Download Hallway for Mac.

Note: If on PC, join the waitlist. We’re actively developing that right now and we’ll send you an email when that’s out after you’ve installed Hallway.

Installing AddRoutes (2:09)

• Next, go to the AddRoutes page and install.
• Install the plug-in inside Blender.
• Connect translation and rotation.

In another video we’ll talk about how to actually wire up blend shapes. The format for getting the information out of Hallway Tile, and into blender is as follows:

• We’re using the address /HT or translation in the X,Y, Z.

One thing to note is that in most games Y is up, so in Blender Z is up a lot of time. What’s going to happen is when we wire this up X, Y, and Z it’s going to map to Hallway’s X, Z, and Y. And rotation, it’s probably going to have some of the same problems, but we’ll deal with that once we get there.

So to get started, I’m going to open up Blender. Got both versions but you’re going to want to make sure that you use Blender 3.0 and we’re going to start with the default scene. In fact, I think what we’re going to do is we’re going to make this box translate and rotate.

Now for step 1.

• Click Edit Preferences.

I’ve already installed AddRoutes but for the sake of completeness, I’m going to remove it and we’re going to install it from scratch.

• Go to Install.
• Find where you downloaded the plugin and you open the zip file. Don’t unzip the plugin.
• Open it up and you’re going to want to make sure that you click the Check Box to enable System AddRoutes.
• After that, it’s very important you want to go down here and say Listen On (located in OSC System Settings) and use the default setting.

If you want to specify a unique IP and port, you can do that. But you’ll have to make sure that you copy this address into Hallway Tile and I’ll show you a little bit about that later. Sometimes you got to restart. I think since I’ve already installed it and uninstalled it, I’m going to be fine. But you’re going to want to restart Blender before you continue.

Setting Up X Location (5:04)

• You’re going to want to click the Cube.
• Over here go to Object Properties.
• Here you can see your transformations, so location X, location Y, location X and rotation in X, rotation Y, rotation Z.
• Right click on any one of the locations and go down to the Option where it says create Real-Time Route.

• Now go over to your panel and you go to the AddRoutes.
• Then, Project Routes. You’re going to see that it created a route for us.
• We’re going to select OSC because that’s the engine that we’re using and remember we’re using that address /HT, so we’re going to say /HT.

A little bit of explanation to what’s actually going on here. So the way this works is that everything in this block is actually referenced by an index. Location X is index 0, location Y is index 1and location Z is index 2, and here you would say this is where you would actually select locations that you wanna map to and from right. So setting this index to 0 is going to map to location x, but if we wanted to map that to location Z, we’d set that to index two and that would map to the Z location.

So we’re going to start with the X. We’ve already specified our address and then down here it’s the same thing, except this is referencing the locations of the translation coming from Hallway. So 0 would be location X, 1 would be location Y coming from Hallway and 2 would be the location of Z coming from Hallway. We want to map X in Hallway to X in Blender. So index 0 to index 0. After that’s done, it’s all set up.

Next step.

• Open up Hallway.
• Go to Customize.
• Go to OSC.
• And you’re going to Enable Streaming.

Now here’s the point at which if you wanted to use a custom IP import. You could specify it here (located in OSC Settings under Host and Port) but just remember in the AddRoutes settings you’re going to want to make sure that this address matches what AddRoutes is listening to.

After that’s done.

• Go back into Blender.
• Click Receive.

Now you can see that as I move my head left and right in the X direction, the cube is also moving left and right in the X direction. That was the first one and that was fairly simple.

Setting Up Y Location (8:23)

Next, let’s go ahead and turn that route off

• I’m going to click over here on Translation.
• We’re going to say Create Real-time Route.
• Same thing as before, OSC is our engine.
• Our Address is /HT.

We’re wanting to map Y so we do index Y or index 1 for location Y inside Blender and then like we talked about before in most games Y is up. With Blender we’re going to reference index 1. In Hallway, we’re going to be referencing index 2 for the Z index, so Y and Z are flipped in Hallway. This might be something that we find a more creative way to simplify later with some sort of transform. But for now, you’re going to want to map the Y translation in Blender to index 2, which maps to the Z location inside Hallway. Click on receive and now as as I move my head further or closer it’s tracking the Z. You can see that updating here.

The other thing to sort of know is that the way we’re doing the calculation of the Y translation, it’s basically your distance in the negative Y direction from your computer screen. So the closer your head gets to your web camera the closer the box is going to get to zero.

Go ahead and zoom in on that box and then if you want to see how that looks with the X, we can enable the X. So as i’m moving my head around we’re getting translations in the X and Y directions.

Setting Up Z Location (11:14)

For the last one. Go ahead and turn these off for now. Turn Enable OSC Streaming OFF. We want to wire up the Z location.

• Create another route same as before engine is OSC.
• Our address is /HT. Inside Blender we want to map index 2 to Hallway’s index, which is actually 1 again because the Y’s and Z’s are flipped.

To do this:

• Go back to Hallway.
• Turn on OSC.
• Hit Receive.

Now, when my head goes up and down, the cube is also going up and down inside Blender. We can go ahead and turn every single one of these on and now we’re getting positional tracking in all three dimensions inside Blender. Turn this off for now (i.e. turn off both Receives located in right panel).

Setting Up X Rotation (12:34)

The last thing to do is connecting the rotations.

Same as before.

• We’re going to select the Box.
• Make sure that we’re in Object Properties.
• Go down to Rotation.
• Right click, Rotation and I’m going to say Create Real-time Route.

Now, the address for rotations is /HR. AddRoutes, it’s smart enough to give us this option to convert from degrees to radians in Hallway. I believe we are sending radians. We want to turn that off.

• Our Engine needs to be OSC.
• Address /HR for rotation.

And let’s just see what index 0to index 0 maps to and make sure Enable OSC Streaming is on (Located in Hallway under OSC Settings).

What you’re seeing here is pitch being tracked, which maps to Blender’s rotation X. Pitch maps to X, so that looks good. Let’s go ahead and turn that one off.

Setting Up Y Rotation (14:34)

• Let’s create Real-time route.
• Engine is OSC that maps to the Y.
• Our address is /HR.

Let’s see, I believe this might be flipped, but let’s turn it on and see what happens. Ah broken. Gotta turn off the degrees to radians calculation, so this one actually is reversed. If you can see this in Hallway I’m turning my head left and right but in reality that’s off again. That’s because the Y and Z axes are flipped so the index is actually going to want it you’re going to want to make the index 2 so that maps properly okay. After that’s done, let’s turn that off.

Setting Up Z Rotation (15:47)

Our last one is to do map roll.

• Create Real-time route.
• OSC in Address /HR we want to map to our Z-axis.
• Want to map to Z axis.
• This one actually is going to be index 1.

We turn it on and now we’re getting the head left and right movement. I can turn all three of these on. Now you can see that as I move my head around, actually I can turn video on real quick. Oh that doesn’t work it’s broken. Um bug report. As I’m moving my head around the cube it’s moving around, putting it all together, we go ahead and enable our translation in X, Y, and Z. There we go. I’m going to move my head around translating and rotating and it’s mapping inside Blender.

Now mapping like sensitivity in units that’s something that we should go over in another video where you would actually instead of controlling a box directly, we’d control an empty object, and then we would bind that empty object using a constraint. Using that constraint we can adjust sensitivity and scale.

That concludes this initial setup video. Thank you!

If you have other questions, you can also get in touch with us on Discord.

Subscribe to our newsletter for more updates on what we’re doing next!

Take Back Your Camera