Highlight Removal In Facial Images

Abstract

We present a method for highlight removal in facial images based on GAN. In contrast to previous works relying on physical models such as dichromatic reflection model, we adopt the structure of CGAN to generate highlight-free images. By taking the facial images with specular highlight as the condition, the network predicts the corresponding highlight-free images. Meanwhile, a novel mask loss is introduced through highlight detection, which aims to make the network focus more on the highlight regions. With the help of multi-scale discriminators, our method generates highlight-free images with high-quality details and fewer artifacts. Moreover, we made a dataset containing both real and synthetic images, and to our best knowledge, this is so far the largest image dataset for facial highlight removal. By comparing with the state-of- the-arts, our method shows high effectiveness and strong robustness in different lighting environments.

Introduction

In this paper, we aim to remove the highlight parts in a facial image through an end-to-end and fully automatic framework. To address the difficulty of lacking in ground truth, we propose a dataset that consists of 150 pairs of real faces and 1170 synthetic faces, where the highlight-free real images are obtained by applying a polarizer on the camera while the synthetic faces are rendered by using graphics models. Our network is pretrained by the synthetic dataset and then finetuned by the real dataset. With the two-step training, we effectively improve the output of the network. To improve the quality of the prediction, we employ the multi-scale discriminators and highlight detection.

Results

Examples of highlight detection. The first row shows source images while the second row shows the detection results. The highlight detection can approximately locate the highlight regions, and the detection results of the generated highlight-free images are all black(no highlight is detected), which also verifies the effectiveness of our method.

Here we show a qualitative comparison between Highlight-net and our method. We present the source facial images as the input of each highlight removal algorithm in the first row (a). In the middle row (b) we provide the results of Highlightnet. In the last row (c) we show the outputs of our method.

Examples of our model being used to remove highlight on faces with different angles and skin color. Rows (a) presents the source facial images with highlight while rows (b) shows the corresponding highlight-free results predicted by our method.

Page updated

Report abuse