Facetorch User Guide

4 min readSep 2, 2022

Introduction

Facetorch is a Python library that can detect faces and analyze facial features using neural networks written in PyTorch. The goal is to gather the best pre-trained face analysis models with open licenses, optimize them for performance using TorchScript, and combine them to create a face analysis tool that one can use for the following:

Face Detection
Facial Representation Learning
Face Verification
Facial Expression Recognition
Deepfake Detection
3D Face Alignment

while having the ability to:

configure using Hydra (OmegaConf)
run reliably with conda-lock and Docker
accelerate on CPU and GPU with TorchScript
extend by adding your FaceDetector, FacePredictor or FaceUtilizer

A list of currently available models can be found in the facetorch GitHub repository.

Tutorial

Docker prerequisite

Using Docker allows you to run facetorch with the least amount of commands and the proper dependencies for your device. Here is a link to get the right Docker version for your platform. To verify Docker installation run this in your terminal:

docker info

Getting facetorch

Facetorch containers can be accessed through DockerHub. A CPU version can be downloaded as follows:

docker pull tomasgajarsky/facetorch:0.1.1

A GPU version build on top of the Nvidia CUDA container is available from the facetorch-gpu DockerHub repository:

docker pull tomasgajarsky/facetorch-gpu:0.1.1

Starting the container

In order to open a command line inside the facetorch image:

CPU version:

docker run -it tomasgajarsky/facetorch:0.1.1 bash

The GPU version requires synchronization between the container and the CUDA device. In this example, the device with index 0 is assigned to the container:

docker run --gpus '"device=0"' -it tomasgajarsky/facetorch-gpu:0.1.1 bash

Running the example

Using the Hydra configuration, the example script initializes the FaceAnalyzer object, runs face analysis models on a test image, and saves the resulting image to the data/output directory.

CPU:

python scripts/example.py

GPU:

python scripts/example.py analyzer.device=cuda

Facetorch will write JSON logs to the console to inform the user of the current state. Models are downloaded from Google Drive only when they are missing.

Output

Image

Facetorch can draw bounding boxes, face indexes (top left corner of each box), and 3D facial landmarks on the input image and save the resulting image to the data/output directory.

Response

The face indexes from the output image corresponding to those of the Response object’s faces. This is how one can obtain the dimensions of the first face:

response.faces[0].dims

Each Face data object contains a dictionary of Predictions under the preds keyword. FacePredictor names in this diagram define preds dictionary keys. In order to get labels for each face from the Facial Expression Recognition model, run the following:

{face.indx: face.preds["fer"].label for face in response.faces}

In a similar way, one can obtain Deepfake Detection labels for test4 image. Configuring the input and output image paths can be done by changing the path_image and path_output arguments in the conf/config.yaml file or by passing them as command line arguments:

python scripts/example.py path_image=data/input/test4.jpg path_output=data/output/test4.png

data/output/test4.png (sideways for testing purposes)

{face.indx: face.preds["deepfake"].label for face in response.faces}

Facial Representation Learning and Face Verification models produce abstract embedding instead of semantic labels. Set include_tensors to True in order to keep the predicted logits in the Response object:

python scripts/example.py path_image=data/input/test3.jpg path_output=data/output/test3.png include_tensors=True

One can compute the similarity between these embeddings to find faces with similar features. In the following example, cosine similarity between face 19 in the top left corner and the rest of the faces in the test3 image is calculated.

import operator
from torch.nn.functional import cosine_similaritybase_emb = response.faces[19].preds["verify"].logitssim_dict = {face.indx: cosine_similarity(base_emb, face.preds["verify"].logits, dim=0).item() for face in response.faces}sim_dict_sorted = dict(sorted(sim_dict.items(), key=operator.itemgetter(1),reverse=True))

Notice that the closest face 9 (bottom middle) has the same facial expression as face 19. Those faces that are the most dissimilar have extreme expressions or are partially covered, for example, 3 (bottom second from right) or 24 (top second from right).

Final words

This guide shows how to run facetorch in a few steps on a CPU or GPU device and simple examples of what data it produces, how to access it, and how to use it.

You might have noticed that not every result was completely accurate. Certain applications of Face Analysis are also controversial, therefore I would like to ask you to use facetorch responsibly with caution and follow the ethics guidelines for Trustworthy AI from European Commission.