PIFuHD — FAIR Neural Network Restores a 3D Human Model

PIFuHD is a neural network architecture for reconstructing a 3D human model from a 2D image. The approach bypasses existing models for the realism of generated 3D models. The model was developed by researchers from Facebook AI Research. PIFuHD is based on the Pixel-Aligned Implicit Function (PIFu) method and a hierarchical layered neural network. The neural network takes into account the global and local contexts of the image, which allows achieving high accuracy of the final 3D model. PIFuHD captures details such as fingers, facial features, and wrinkles in the person’s clothing. Previous approaches were not capable of such detail.

Source: Facebook AI

What is the problem

Due to limitations in the memory of current electronic devices, past approaches more often took a compressed image as input. However, they produced less accurate predictions or predictions at low resolution. Researchers work around this limitation by using a two-tier architecture in PIFuHD. The model takes into account the global and local contexts.

The approach architecture

The neural network accepts an image of a person with a resolution of 1024 × 1024 as input. At the output, the approach gives a 3D model of the person. The architecture of the method consists of two levels of PIFu modules:

  1. A basic level that focuses on extracting global features from an image. This module is similar to PIFu;
  2. A refinement level that focuses on extracting local context information and adding precise detail to the 3D model
Visualization of the components of the approach. Source: Facebook AI

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Mikhail Raevskiy

Mikhail Raevskiy

430 Followers

Bioinformatician at Oncobox Inc. (@oncobox). Research Associate