GFPGAN: A Machine Learning Model for Enhancing the Quality of Facial Images

This is an introduction to「GFPGAN」, a machine learning model that can be used with ailia SDK. You can easily use this model to create AI applications using ailia SDK as well as many other ready-to-use ailia MODELS.

Overview

GFPGAN (Generative Facial Prior GAN) is a machine learning model announced by Tencent in January 2021 that takes low-quality facial images as input and restores them to high-quality facial images.

GitHub — TencentARC/GFPGAN: GFPGAN aims at developing Practical Algorithms for Real-world Face…

Towards Real-World Blind Face Restoration with Generative Facial Prior

Architecture

GFPGAN belongs to the field of Blind Face Restoration, a task that involves restoring deterioration such as low resolution, noise, blur, and compression noise.

GFPGAN corrects facial details and color in a single pass.

GFPGAN architecture (Source: https://arxiv.org/pdf/2101.04061.pdf)

GFPGAN starts by detecting the face using RetinaFace, which return 5 landmarks delimiting the face area on which to perform an affine transformation using iterative optimization with cv2.LMEDS through cv2.estimateAffinePartial2D to normalize the orientation of the face.

Input image (Source: https://github.com/TencentARC/GFPGAN/blob/master/inputs/whole_imgs/10045.png)

Normalized face

Noise reduction is performed on each face using UNet, then StyleGAN2 is applied to restore the images to high-resolution facial pictures.

The input for GFPGAN is a 512x512 image in RGB order with values in the range [-1, 1].

Input image

The output image has the same resolution, and it is transformed back using inverse affine transformation, and a Gaussian blur is applied to the edges before blending with the original image.

Mask for Boundary Blending

Output image

GFPGAN results (Source: https://arxiv.org/pdf/2101.04061.pdf)

Usage with Stable Diffusion

StableDiffusionWebUI comes standard with GFPGAN integrated. It can be activated from the settings, under the “Face restoration” parameters.

Usage with ailia SDK

GFPGAN can be used with ailia SDK using the following command.

$ python3 gfpgan.py --input input.jpg --savepath output.jpg

By adding the -m option, it is possible to select the version of GFPGAN to use. The default version is v1.3.

$ python3 gfpgan.py --input input.jpg --savepath output.jpg -m v1.4

Here is a comparison of the result between version 1.3 and 1.4

ailia-models/generative_adversarial_networks/gfpgan at master · axinc-ai/ailia-models

Usage with Unity

There is a sample available for using GFPGAN from Unity. In this sample, BlazeFace is used instead of RetinaFace.

ailia-models-unity/Assets/AXIP/AILIA-MODELS/GenerativeAdversarialNetworks at master ·…

Input

Output

ax Inc. has developed ailia SDK, which enables cross-platform, GPU-based rapid inference.

ax Inc. provides a wide range of services from consulting and model creation, to the development of AI-based applications and SDKs. Feel free to contact us for any inquiry.