Face Hallucination

(Face Super-Resolution)

Input (Low Resolution Image) => Output (Super Resolved Image)

Problem Statement

Low-resolution face images affects the accuracy of face recognition and face verification. So, Super-resolved output of these low-resolution is the solution for enhancing the face recognition and face verification accuracy.

Introduction

Face super-resolution, also known as face hallucination, which is aimed at enhancing the resolution of low-resolution (LR) face image to generate the corresponding high-resolution (HR) face image, is a domain-specific image super-resolution problem. Generating high-resolution image is of great importance for many practical face applications, such as face recognition and face verification. Since the structure of the face image is complex and sensitive, obtaining a super-resolved face image is more difficult than generic image super-resolution.

Recently, with great success in the high-level face recognition task, deep learning methods, especially generative adversarial networks (GANs), have also been applied to the low-level vision task – face hallucination. Recently, face super-resolution has received considerable attention with deep learning techniques and it has always been a hot topic since its birth in the field of image processing and computer vision society.

Pipeline / Flowchart

Applications

Face Video Super-resolution.
Old Photo Restoration.
Face 3D Reconstruction.
Improved Face Recognition.
Person Identification.

Image Quality Assessment Techniques (Evaluation Criteria)

Quantitative (Statistical) Evaluation

Mean Square Error (MSE):

MSE represents the cumulative squared error between the LR image and HR image.

Peak Signal-to-Noise Ratio (PSNR):

PSNR focuses on distance between every pair pixel in two images (LR and HR).

Structural Similarity Index Measure (SSIM):

SSIM assesses similarity between luminance, contrast and structure.

Multi-Scale Structural Similarity Index Measure (MS-SSIM):

SSIM is less reliable than assessing images locally. Thus, MS-SSIM is proposed to enhance reliability, which divides the whole images into multiple windows, and first assesses SSIM for every window separately, and then converges them to obtain the final MS-SSIM.

Qualitative (Perceptual) Evaluation

Frechet Inception Distance score (FID):

FID compares the difference between feature maps.

Learned Perceptual Image Patch Similarity (LPIPS):

LPIPS evaluate the distance between image patches.

Perception based Image Quality Evaluator (PIQE):

PIQE score is inversely correlated to the perceptual quality of an image.

Naturalness Image Quality Evaluator (NIQE):

NIQE measures the distance between the NSS-based features calculated from image A to the features obtained from an image database used to train the model.

Research Gap / Challenges / Future Directions

High-dimensional Facial Prior Information:

Face super-resolution methods is increasingly complex and has higher than higher dimensions, from 2D images (facial landmarks, facial heatmaps, parsing maps) to 3D prior, which means higher-dimensional prior provides richer information. Thus, higher-dimensional prior can significantly enhance face reconstruction.

Balance between Subjective and Objective Quality:

Face super-resolution tends to recover SR with higher PSNR but poorer visual quality and vice versa. However, the balance between subjective and objective quality is important and existing face super-resolution methods ignore how to find a balance between them.

Lightweight Face Super-resolution Models:

Deep learning-based face super-resolution methods have achieved great breakthroughs, they have difficulty in deploying real-world applications, which is caused by a mass of parameters and high computation cost. Hence, developing models with more lightweight and lower computation cost is still a major challenge.

More Challenging Scales:

More challenging scales, such as x32, x64, can be explored. Existing face super-resolution methods mainly focus on the case of the magnification factors x8.

Unsupervised or Self-supervised Methods:

Unsupervised or self-supervised methods will become mainstream face super-resolution methods. Due to the observation that the degradation process in the real world is too complex to be simulated.

Related Datasets

Download Links:

Cross-Age Celebrity Dataset (CACD)

Celebrities in Frontal-Profile (CFP)

Labeled Faces in the Wild (LFW)

CelebFaces Attributes (CelebA)

Flickr-Faces-HQ (FFHQ)

Relevant Study

Face super-resolution using WGANs

This video is about Applications of face super-resolution and GANs.

Survey Papers:

Jiang, J., Wang, C., Liu, X. and Ma, J., 2021. Deep Learning-based Face Super-resolution: A Survey. arXiv preprint arXiv:2101.03749.
Liu, H., Zheng, X., Han, J., Chu, Y. and Tao, T., 2019. Survey on GAN-based face hallucination with its model development. IET Image Processing, 13(14), pp.2662-2672.

Journal Papers:

Huang, B., Chen, W., Wu, X., Lin, C.L. and Suganthan, P.N., 2018. High-quality face image generated with conditional boundary equilibrium generative adversarial networks. Pattern Recognition Letters, 111, pp.72-79.
Yang, X., Lu, T., Wang, J., Zhang, Y., Wu, Y., Wang, Z. and Xiong, Z., 2018, September. Enhanced discriminative generative adversarial network for face super-resolution. In Pacific Rim Conference on Multimedia (pp. 441-452). Springer, Cham.
Ataer-Cansizoglu, E., Jones, M., Zhang, Z. and Sullivan, A., 2019. Verification of very low-resolution faces using an identity-preserving deep face super-resolution network. arXiv preprint arXiv:1903.10974.
Hsu, C.C., Lin, C.W., Su, W.T. and Cheung, G., 2019. Sigan: Siamese generative adversarial network for identity-preserving face hallucination. IEEE Transactions on Image Processing, 28(12), pp.6225-6236.
Li, J., Zhou, Y., Ding, J., Chen, C. and Yang, X., 2020. ID Preserving Face Super-Resolution Generative Adversarial Networks. IEEE Access, 8, pp.138373-138381.
Yu, X., Porikli, F., Fernando, B. and Hartley, R., 2020. Hallucinating unaligned face images by multiscale transformative discriminative networks. International Journal of Computer Vision, 128(2), pp.500-526.
Wang, M., Chen, Z., Wu, Q.J. and Jian, M., 2020. Improved face super-resolution generative adversarial networks. Machine Vision and Applications, 31(4), pp.1-12.
Chen, C., Gong, D., Wang, H., Li, Z. and Wong, K.Y.K., 2020. Learning Spatial Attention for Face Super-Resolution. IEEE Transactions on Image Processing, 30, pp.1219-1231.
Zhang, M. and Ling, Q., 2020. Supervised Pixel-Wise GAN for Face Super-Resolution. IEEE Transactions on Multimedia.
Zhang, Y., Tsang, I.W., Li, J., Liu, P., Lu, X. and Yu, X., 2021. Face hallucination with finishing touches. IEEE Transactions on Image Processing, 30, pp.1728-1743.
Yu, X., Fernando, B., Hartley, R. and Porikli, F., 2019. Semantic face hallucination: Super-resolving very low-resolution face images with supplementary attributes. IEEE transactions on pattern analysis and machine intelligence, 42(11), pp.2926-2943.
Grm, K., Scheirer, W.J. and Štruc, V., 2019. Face hallucination using cascaded super-resolution and identity priors. IEEE Transactions on Image Processing, 29, pp.2150-2165.
Liu, Z.S., Siu, W.C. and Chan, Y.L., 2021. Features Guided Face Super-Resolution via Hybrid Model of Deep Learning and Random Forests. IEEE Transactions on Image Processing, 30, pp.4157-4170.
Zhang, Y., Tsang, I., Luo, Y., Hu, C., Lu, X. and Yu, X., 2021. Recursive Copy and Paste GAN: Face Hallucination from Shaded Thumbnails. IEEE Transactions on Pattern Analysis and Machine Intelligence.
Yu, X., Zhang, L. and Xie, W., 2021. Semantic-Driven Face Hallucination Based on Residual Network. IEEE Transactions on Biometrics, Behavior, and Identity Science, 3(2), pp.214-228.
Jiang, K., Wang, Z., Yi, P., Lu, T., Jiang, J. and Xiong, Z., 2020. Dual-path deep fusion network for face image hallucination. IEEE Transactions on Neural Networks and Learning Systems.

Conference Papers:

Yu, X. and Porikli, F., 2016, October. Ultra-resolving face images by discriminative generative networks. In European conference on computer vision (pp. 318-333). Springer, Cham.
Yu, X., Fernando, B., Hartley, R. and Porikli, F., 2018. Super-resolving very low-resolution face images with supplementary attributes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 908-917).
Chen, Y., Tai, Y., Liu, X., Shen, C. and Yang, J., 2018. Fsrnet: End-to-end learning face super-resolution with facial priors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2492-2501).
Yu, X., Fernando, B., Ghanem, B., Porikli, F. and Hartley, R., 2018. Face super-resolution guided by facial component heatmaps. In Proceedings of the European conference on computer vision (ECCV) (pp. 217-233).
Zhang, K., Zhang, Z., Cheng, C.W., Hsu, W.H., Qiao, Y., Liu, W. and Zhang, T., 2018. Super-identity convolutional neural network for face hallucination. In Proceedings of the European conference on computer vision (ECCV) (pp. 183-198).
Lu, Y., Tai, Y.W. and Tang, C.K., 2018. Attribute-guided face generation using conditional cyclegan. In Proceedings of the European conference on computer vision (ECCV) (pp. 282-297).
Lee, C.H., Zhang, K., Lee, H.C., Cheng, C.W. and Hsu, W., 2018. Attribute augmented convolutional neural network for face hallucination. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 721-729).
Kim, D., Kim, M., Kwon, G. and Kim, D.S., 2019. Progressive face super-resolution via attention to facial landmark. arXiv preprint arXiv:1908.08239.
Kazemi, H., Taherkhani, F. and Nasrabadi, N.M., 2019, September. Identity-aware deep face hallucination via adversarial face verification. In 2019 IEEE 10th International Conference on Biometrics Theory, Applications and Systems (BTAS) (pp. 1-10). IEEE.
Bayramli, B., Ali, U., Qi, T. and Lu, H., 2019, December. FH-GAN: Face hallucination and recognition using generative adversarial network. In International Conference on Neural Information Processing (pp. 3-15). Springer, Cham.
Dou, H., Chen, C., Hu, X., Xuan, Z., Hu, Z. and Peng, S., 2020, October. PCA-SRGAN: Incremental Orthogonal Projection Discrimination for Face Super-resolution. In Proceedings of the 28th ACM International Conference on Multimedia (pp. 1891-1899).
Yang, L., Wang, S., Ma, S., Gao, W., Liu, C., Wang, P. and Ren, P., 2020, October. Hifacegan: Face renovation via collaborative suppression and replenishment. In Proceedings of the 28th ACM International Conference on Multimedia (pp. 1551-1560).