Face Video Deblurring using 3D Facial Priors

Wenqi Ren¹, Jiaolong Yang², Senyou Deng¹, David Wpif², Xiaochun Cao¹, and Xin Tong²

¹SKLOIS, IIE CAS, ²Microsoft Research

Abstract

Existing face deblurring methods only consider single frames and do not account for facial structure and identityinformation. These methods struggle to deblur face videos that exhibit significant pose variations and misalignment. In this paper we propose a novel face video deblurring network capitalizing on 3D facial priors. The model consists of two main branches: i) a face video deblurring sub-network based on the encoder-decoder architecture, and ii) a 3D face rendering branch for predicting 3D priors of salient facial structures and identity knowledge. These structures encourage the deblurring branch to generate sharp faces with detailed structures. Our method not only uses low-level information (i.e., intensity similarity), but also middle-level information (i.e., 3D facial structure) and high-level knowledge (i.e., identity content) to further explore spatial constraints of facial components from blurry face frames. Extensive experimental results demonstrate that the proposed algorithm performs favorably against the state-of-the-art methods.

Figure 1. The proposed face video deblurring architecture. Our model consists of two streams, the top green block is a ResNet-50 network which aims to restore 3D face coefficient vector x = (α; β; δ; γ; p) 2 R239 and render a sharp face capitalizing on the coefficient vector. The bottom orange block focuses on face deblurring guided by the extracted identity vector α and the rendered sharp face structure from the facial structure and identity extraction branch.

Paper

[Paper]

[Supp]

Code

[GitHub]

Citations

Please cite this paper in your publications if it helps your research:

@inproceedings{Ren-ICCV-2019,

author = {Ren, Wenqi and Yang, Jiaolong and Deng, Senyou and Wipf, David and Cao, Xiaochun and Tong, Xin},

title = {Face Video Deblurring using 3D Facial Priors},

booktitle = {IEEE Internatial Conference on Computer Vision},

year = {2019}

}