3D Face Modeling and Reconstruction

ABSTRACT

This tutorial focuses on the problem of reconstructing a 3D model of a human face from a single image, possibly captured in unconstrained conditions, i.e., in the wild . Faces represent a particular category of objects with specific properties that can be leveraged to address the reconstruction and modeling problems exploiting dedicated techniques, the most widespread being the 3D Morphable Model (3DMM). Since its advent, the 3DMM has gained attraction because of its simplicity and effectiveness in many applications; several variants have been proposed across the years, making the 3DMM an actual foundation in the field. Recent technical advancements and the increased availability of 3D data make the 3DMM still an active and evolving research direction, for which the interest is not diminished. All the above makes the 3DMM and its recent intersection with deep learning mechanisms a valuable and active topic in the field of 3D vision that interested researchers should know. The tutorial aims at providing the fundamental knowledge for constructing a morphable model of the face and how to use it for a variety of applications.

Brief overview

Basics

In the first part of the tutorial, we will provide the basic concepts and definitions of the 3D Morphable Model. We will then introduce the problem of 3D dense registration of point clouds, which is itself a challenging and open problem in many applications, representing a necessary prerequisite for building a training set for the 3DMM. Then, we will present the optimization techniques used to estimate the 3DMM parameters ( fitting) from a single image and illustrate how the fitting permits the reconstruction of an approximated and smooth foundation shape of the face. Ultimately, techniques for manipulating the reconstructed shape and combining it with texture data will be shown so as to render realistic images of the face.

Recent Trends

In the second part, we will introduce the recent advances in the field obtained by deploying the power of deep learning techniques. In general, the advent of such techniques has drastically changed the way computer vision problems are being addressed. Even though the diffusion of such techniques in the 3D vision field has had a slower expansion because of the diverse data representation (irregular 3D data against regular RGB imagery), deep learning techniques are currently being applied with promising results to 3D data represented as depth images as well. We will focus on the applications of the 3DMM and introduce novel deep learning-based techniques for reconstructing, manipulating and generating 3D faces.