Written Guide
Last Updated: Aug 2024
Introducing FaceTracker for Blender.
Get easy access to professional face tracking with custom head meshes generated by the built-in FaceBuilder module right inside of your favourite 3D creation suite. Track faces and seamlessly enhance them with 3D visual effects and simulations — all within Blender.
Installation
Blender 4.2: Download KeenTools Blender Pack, open Blender Preferences > Add-ons, click on the little arrow in the top right corner and select Install from Disk. Navigate to the downloaded zip file and install it.
Next, install Core Library. You can do it either online or offline.
Unfold KeenTools extension and agree to the Licence Agreement. If your machine is connected to the internet, simply press Install online. Core Library will be downloaded and installed automatically.
For offline installation, download Core Library manually from our website. After download, press Install from disk and navigate to the downloaded file. Please keep in mind that the add-on and the core library versions should match.
If you already have any other KeenTools add-on installed in Blender, just update it. The new FaceTracker tab will appear on the sidebar (Shortcut: N — hide/show).
Inputs
Start by pressing N to view the sidebar and then click on the FaceTracker tab to open the main panel. Press Create new FaceTracker. This will open the main panel with Inputs on the top. The fields marked in red are empty fields that must be filled to proceed to tracking.
Camera
By default, FaceTracker picks up the Camera from your Blender scene. If you accidentally delete it, you can create a new camera by clicking Add > Camera in the top left of the viewport.
You can plug a new camera into the Camera Input from the dropdown list or directly from the viewport with the picker (Shortcut: E).
Clip
Click on the folder icon next to the Clip field and load your footage. This can be either an image sequence or a movie file.
For faster tracking you will need to analyse the clip. Press Analyse, select the frame range and hit OK. FaceTracker will scrub through the timeline forward and backwards and create an advanced motion vector file (.precalc ) on your disk, which you can use later for tracking other objects in the same shot. By default, it will be saved in your temporary folder on MAC or disk C on Windows. But, of course, you can choose a different location.
Head
You will need a face geometry with FaceBuilder topology to track a face in the clip. You can either plug an existing FaceBuilder head in here or create a new one on-the-go by pressing New.
Create Face Geometry
Once you press New, you jump into the face building part of the pipeline. The built-in FaceBuilder module allows you to create custom face geometry based on snapshots of video frames and image files.
Take a snapshot of a video frame
Scroll through the timeline and choose a frame in which the actor’s face is seen clearly and looks most three-dimensional. A ¾ view usually works best. Once you’ve selected the frame, press Take Snapshot.
Customise your face mesh
FaceBuilder automatically detects the face and matches it with the background image. Although, sometimes it may look like a perfect match, it is still a rough estimation of the face shape which needs to be adjusted manually by dragging pins (red dots) in the viewport to line up the facial features. We suggest first deleting (right-click) the pins whose position seems irrelevant. Create and drag (left-click) new pins until the mesh is by and large lined up.
Try to keep the number of pins as low as possible. Sometimes deleting a pin works better than creating a new one.
The tracking accuracy will depend on the accuracy of your 3D model. You can improve your face mesh by adding more views. Press + snapshot to choose another frame with a different enough angle. Or add images from files by clicking on + img file. The most valuable angles for face building are: ¾, profile and front. For each new view a new camera will be automatically added to your scene and a new view button will appear in the Views tab. You can switch between the views simply by clicking on them. If you need to delete a view, click on the hamburger button on the right and select Delete.
Line up face mesh in each of the newly added views. We suggest using the Auto Alight option first, and then dragging the pins in the viewport until everything is lined up correctly.
FaceBuilder module is a seamless integration of FaceBuilder inside FaceTracker. You can actually go over to the FaceBuilder tab and ‘sculpt’ your face geometry from there, as well as get access to appearance settings (wireframe colour and opacity) in the Appearance tab. All changes you make in the integrated module are automatically applied in the FaceBuilder add-on and vice versa.
Once you’re finished face building, click on Back to FaceTracker. Before you move on to tracking, you can check how your model looks in 3D and whether you’re happy with it. If you discover anything weird or positively wrong, you can always press Edit and fix that by going through the views, removing wrongly placed pins, lining up things that are off, removing or adding new views.
Learn about creating high-precision 3D heads with FaceBuilder in our detailed video tutorial and written guide.
Tracking
The actual tracking consists of the following steps: model placement, tracking and refining.
Model Placement
Press Start Pinmode in the Tracking tab. What you need to do now is overlay your face geometry on top of the face in the initial frame. This does not necessarily have to be the first frame of your clip. Find a frame in which the face is seen clearly and has an expression most close to neutral. Press Auto Align for automatic model placement. Click and drag pins to match the mesh with the face. Left-click anywhere on the mesh to create new pins. Right-click on the pins you want to remove.
Note that pins are only used for adjusting the model’s position. Higher number of pins will not improve your tracking. What matters is the accuracy of the model placement.
When you press Auto Align or adjust mesh position, FaceTracker creates a new manual keyframe (green dotted line on the timeline) which stores the model position, rotation and facial expression in the current frame.
Wireframe colour and opacity are adjustable, as are the size and sensitivity of the pins. You will find the basic settings in the Appearance tab. More settings in Preferences > Add-ons.
Camera setup
The key to successful model placement and perfect tracking results is using correct camera settings. If you know the focal length and film size of the camera on the set, just type them in in the dedicated Camera tab. In case you don’t, or unsure about your camera data, you can tick Estimate focal length checkbox to get that value calculated for you. Note that automatic focal length estimation works only for the current frame while you’re pinning the model, and you need to have at least 4 pins set on the wireframe to activate it.
Track & Refine
Select tracking mode depending on your needs:
Head — facial performance + head motion.
Camera — facial performance + camera track.
Click on the Track Forward button. You’ll see FaceTracker estimating head position and facial expression in the consequent frames. Whenever you see the mesh is off, pause tracking and adjust its position and facial expression manually using the existing pins or creating new if necessary. Every time you make manual adjustments FaceTracker will create a new manual keyframe and store the new position, rotation and facial expression data in it.
But the range between the manual keyframes needs to be updated too. You can update it automatically by pressing Refine. Keep on tracking and refining until you’re satisfied with your face track.
You can also track backwards or frame by frame using the tracking buttons in the Tracking tab.
The resulting face track is a combination of head movement in 3D space relevant to camera (or camera relevant to head, when Camera tracking mode is selected), and blendshapes animation representing the facial performance. The exact location and rotation of the 3D head in each frame can be seen in the Item tab. The facial animation is stored as blendshapes and can be accessed through the Shape Keys tab in Object Data Properties.
Make it Stable!
Improve tracking accuracy with Lock View
Sometimes it’s hard to assess the tracking quality especially in the dynamic shots. Use the Lock View button or keyboard shortcut L to stabilize the clip in the viewport. With no pins selected, FaceTracker will snap the view to your Head geometry. But you can also lock the view to a specific point on your 3D model by selecting one or a group of pins. Learn more about this feature in our video tutorial.
Masking
FaceTracker offers three types of masks to deal with almost any occlusion you might encounter: Surface, Compositing, and Sequence masks.
Surface mask allows you to select polygons right on the surface of the face geometry to exclude them from tracking. This can be useful when dealing with accessories like glasses, hats or hair falling in the face by taking out the parts of the face mesh interacting with them.
To apply a Surface mask, switch to Edit Mode and select the polygons you want to mask out, then go to Object Data Properties, create a new vertex group by clicking on +, and press the Assign button below. Return to Object Mode and select the newly created group in the Surface mask dropdown list. You can then switch to Pinmode to view the mask.
Another way of dealing with occlusions is 2D masking, which works for most overlays like the actor’s hand or props getting in the line of sight. You can either draw a 2D mask right in Blender using the Compositing mask option or import a pre-made mask using the Sequence option. Note that you can’t select both Compositing and Sequence masks within one tracking session. You can only use one at a time.
In order to use a Compositing mask, go to Movie Clip Editor and select your clip. Switch the Tracking mode to Mask and click on Add > Circle (or other shapes that fit your situation). You can now press Move and place your mask over the element you want FaceTracker to ignore when tracking. Learn more about editing masks in this Blender manual.
You can adjust your mask position in other frames by creating new keyframes. Note that the keyframes you create in the Movie Clip Editor are independent of those generated by FaceTracker. Once you’re happy with the mask shape and position, go back to the 3D viewport and select your mask in the Compositing mask tab. Go to Pinmode to see your mask in action.
If you have a pre-made mask, you can import it using the Sequence mask option. You can then choose which of the RGBA channels you want to use for masking.
Scene Tab
You can easily manipulate your scene from the Scene tab on the FaceTracker panel. The Transform section lets you Scale and Position your scene with the two corresponding buttons. You can scale either the whole scene relative to selected Pivot point or Geometry relevant to Camera, or Camera relevant to Geometry. And when doing the latter, you can also lock the camera visualisation scale.
You can select either Camera or Geometry as the pivot point to position your scene in 3D space. You can set the Location and Rotation values manually or use preset positions such as World Origin and 3D Cursor.
Go to the Animation section when you need to convert your Camera animation to Head or vice versa. Note that if both the Camera and Head are animated, pressing the Convert button will convert all animation to one, and then this operation will not be reversible!
Press Bake if you need to unparent Camera and/or face geometry. This will bake them, along with their animation, to World space.
Important! If your model rotates more than 360° between certain keyframes, we recommend using the Unbreak Rotation option. The algorithm behind it will go through all keyframes and make rotation continuous thus avoiding 360 degree jumps. It means that if rotation was 355° in the keyframe #24 and 10° in the keyframe #25, pressing this button will add 360° to keyframe #25, so the resulting rotation of the model will be 370°. It works backwards as well: going from 10° to 355° will make it -5°.
You need to be careful if your model rotates both forward and backwards between the nearest keyframes — then this operation will most likely make things worse.
Texturing
The Texture tab lets you create an animated texture map or texture blended from selected frames using background video as source.
When in Pinmode, unfold the Texture tab, scroll through the timeline and add source frames by pressing + next to the Add frames field. Use — to remove a frame from the list.
For best results, choose frames where the face is clearly seen from different enough angles. Avoid frames that look blurry and dark.
As you click Create Texture, FaceTracker projects the texture from the selected frames and bakes it onto your face geometry. Click the gear icon to access Texturing settings. Here, you can adjust the size of your texture and also set up things like Angle Strictness, allowing you to control how much the pixel colour will be affected depending on the viewing angle, and Expand Texture Edges which lets you cover dark spots on the texture by expanding the edges.
Click Export to get your texture as a PNG, JPEG, or EXR file.
Press Save to Sequence to export an animated texture map.
Exporting results
Use the Export button in the Animation section of the Scene tab to export your tracking results. Specify the export option using the dropdown menu right above:
Head — head animation
Camera — camera animation
Selected pins — one or multiple Empties (drop a pin on the mesh and set its orientation)
FACS as CSV — facial animation as FACS in a .csv file (Facial Action Coding System)
Check Linked to keep exported animation synced with further changes in the tracking data.
Video tutorials
Check out these video guides to learn more about FaceTracker for Blender.
More tutorials are on their way!
Follow us: Facebook, Linkedin, Twitter, Instagram, YouTube, Discord, Reddit
