Introduction

• Face privacy is essential because facial recognition (FR) technology can covertly identify individuals from images or video streams, creating major privacy issues. As FR technology progresses, it increases the risk for individuals who are unknowingly recorded in public spaces, with potential security vulnerabilities or data breaches potentially exposing sensitive information. International regulations, such as GDPR, highlight the need to protect personal data, including facial images, to prevent misuse and unauthorized access. Protecting face privacy involves adopting measures like anonymization, where identifiable facial features are obscured or removed, thereby protecting individual identities and adhering to privacy laws.

Toward a Privacy-Preserving Face Recognition System: A Survey of Leakages and Solutions

• Recent advancements in face recognition (FR) technology in surveillance systems make it possible to monitor a person as they move around. FR gathers a lot of information depending on the quantity and data sources. The most severe privacy concern with FR technology is its use to identify people in real-time public monitoring applications or via an aggregation of datasets without their consent. Due to the importance of private data leakage in the FR environment, academia and business have given it a lot of attention, leading to the creation of several research initiatives meant to solve the corresponding challenges. As a result, this study aims to look at privacy-preserving face recognition (PPFR) methods. We propose a detailed and systematic study of the PPFR based on our suggested six-level framework. Along with all the levels, more emphasis is given to the processing of face images as it is more crucial for FR technology. We explore the privacy leakage issues and offer an up-to-date and thorough summary of current research trends in the FR system from six perspectives. We also encourage additional research initiatives in this promising area for further investigation. 

• Our proposed architecture for a CCTV surveillance facial recognition (FR) system is derived from the literature and includes a comprehensive system diagram. It also identifies potential privacy breaches or attack types, categorized by attack surfaces such as the camera, image, face, model, database, or communication points.

• To address privacy at both the image and face levels, we have developed various methods, including latent space navigation and machine unlearning. These techniques allow us to learn, remove, or alter identifiable features related to a person's identity, thereby protecting individual privacy. By manipulating the latent space of facial data, we can effectively anonymize or de-identify individuals in images and video frames, ensuring that their personal information is safeguarded while still enabling the use of facial recognition technology for legitimate purposes.

High-Quality Face Caricature via Style Translation

• Abstract: Caricature is an exaggerated form of artistic portraiture that accentuates unique yet subtle characteristics of human faces. Recently, advancements in deep end-to-end techniques have yielded encouraging outcomes in capturing both style and elevated exaggerations in creating face caricatures. Most of these approaches tend to produce cartoon-like results that could be more practical for real-world applications. In this study, we proposed a high-quality, unpaired face caricature method that is appropriate for use in the real world and uses computer vision techniques and GAN models. We attain the exaggeration of facial features and the stylization of appearance through a two-step process: Face caricature generation and face caricature projection. The face caricature generation step creates new caricature face datasets from real images and trains a generative model using the real and newly created caricature datasets. The Face caricature projection employs an encoder trained with real and caricature faces with the pre-trained generator to project real and caricature faces. Using the encoder and generator’s latent space, we perform an incremental facial exaggeration from the real image to the caricature faces. Our projection preserves the facial identity, attributes, and expressions from the input image. Also, it accounts for facial occlusions, such as reading glasses or sunglasses, to enhance the robustness of our model. Furthermore, we comprehensively compared our approach with various state-of-the-art face caricature methods, highlighting our process’s distinctiveness and exceptional realism.

• We present the overview of our proposed caricature creation method. Our method consists of two key steps: Face Caricature Generation and Face Caricature Projection. In the first step, Face Caricature Generation, we create a caricature dataset from real faces. A StyleGAN is trained with real and caricature faces, which can produce face caricatures and real images with different styles. The second step, Face Caricature Projection, involves training an encoder using the pre-trained StyleGAN. (a) The encoder training process uses real and newly created caricature faces. (b) The first row shows the incremental facial exaggeration from real to caricature faces, and the second row shows the style change with facial exaggeration.

• We visualize the results obtained by the caricature creation method. (a) The input real face, (b) our projected caricature face, and mixed style caricature faces.

Face De-Identification using Face Caricature

• Abstract: Face privacy concerns revolve around the ethical, social, and technological implications of collecting, storing, and using facial data. With the advancement of deep learning techniques, realistic face privacy involves techniques that obscure or alter facial features effectively without compromising the usability or quality of the visual content. Modern face privacy techniques suffer from three main problems: (1) lack of human perception, (2) indistinguishability, and (3) loss of facial attributes. Modern face privacy techniques generate random, realistic faces to conceal the identifiable features of the original faces but lack the application of human perception to face de-identification. Indistinguishability arises with the highly realistic nature of fake faces used in face privacy, making it difficult to distinguish whether a face has been manipulated. Most face-privacy methods also fail to retain the facial attributes of the de-identified faces. Our face de-identification method is designed to address all three issues mentioned. We propose a novel face de-identification method that considers both human perception and face recognition models when de-identifying a face. We explore the tradeoff between a user misidentifying the original identity with a well-known celebrity and a facial recognition model that tries to identify the original identity. We generate caricature faces of the de-identified faces to ensure our manipulated faces can be distinguished effortlessly. The face caricatures are the exaggeration of the eyes and mouth region, and we provide different exaggeration scales depending on preference and application. We perform an attribute preservation optimization process to retrieve all the facial attributes. We demonstrated our method through a series of both qualitative and quantitative experiments with numerous user studies.

• Overview of our de-identification framework with face caricatures. By optimizing latent code, we perform de-identification using a target celebrity face utilizing a pre-trained encoder and a StyleGAN. We generate caricatures from the incrementally de-identified faces as our final result for privacy protection.

• We visualize our face de-identification with different caricature results. (a) Input face, (b) target celebrity face, (c) de-identified face [α = 1], (d) Our caricature de-identified face [α = 1] with different scale → [small; medium; large], (e) de-identified face [α = 0], (f) Our caricature de-identified face [α = 0] with different scale → [small; medium; large].

Diffusion Based Identity Removal

• With machine unlearning becoming increasingly important, our approach focuses on selectively removing specific identities from a pre-trained diffusion model and refining pre-trained models without the need to train from scratch. Our Identity Conditional Diffusion Model (ID Conditional DM) precisely eliminates unwanted identities while maintaining other important features and generating images associated with the target identity. Moreover, our method provides clear visual insights into the unlearning process, demonstrating its efficacy and the underlying mechanisms that facilitate the selective removal of identity features. This contributes to a more secure and privacy-conscious framework in machine learning applications, offering a practical solution for managing sensitive information. We are currently working on diffusion-based identity removal.

Papers

• Lamyanba Laishram, Muhammad Shaheryar, Jong Taek Lee, Soon Ki Jung, Toward a Privacy-Preserving Face-Recognition System: A Survey of Leakages and Solutions, ACM Computing Surveys [Just Accepted]

• Lamyanba Laishram, Jong Taek Lee, Soon Ki Jung, Face De-Identification using Face Caricature,  IEEE Access, Vol. 12, pp. 19344-19354, ISSN. 2169-3536, 2024 (2024.01.22), JCR : 54.1(Q2)

• Muhammad Shaheryar, Jun Hyeok Jang, Jong Taek Lee and Soon Ki Jung, Targeted Forgetting Noise-Aided Machine Unlearning with Deep Feature Visualization, The International Workshop on Frontiers of Computer Vision (IW-FCV), (2024.02.19 ~ 2024.02.21)

• Lamyanba Laishram, Muhammad Shaheryar, Jong Taek Lee, Soon Ki Jung, High-Quality Face Caricature via Style Translation, IEEE Access, Vol. 11, pp.138882-138896, ISSN.2169-3536, 2023 (2023.12.07), JCR : 54.1(Q2)

• Muhammad Shaheryar, Lamyanba Laishram, Jun Hyeok Jang, Jong Taek Lee and Soon Ki Jung, Learn to Unlearn: Targeted Unlearning in ML, 6th International Conference on Culture Technology (ICCT 2023), (2023.12.1 ~ 2023.12.4)

• Muhammad Shaheryar, Lamyanba Laishram, Jong Taek Lee and Soon Ki Jung, Latent Space Navigation for Face Privacy: A Case Study on the MNIST Dataset, The 18th International Symposium on Visual Computing (ISVC) (2023.10.16 ~ 2023.10.18)

• Lamyanba Laishram, Muhammad Shaheryar, Jong Taek Lee and Soon Ki Jung, A Style-based Caricature Generator, The 29th International Workshop on Frontiers of Computer Vision (IW-FCV2023), (2023.02.20 ~ 2023.02.23)

• Muhammad Shaheryar, Lamyanba Laishram, Jong Taek Lee and Soon Ki Jung, Multi-Attributed Face Synthesis for One-Shot Deep Face Recognition, The 29th International Workshop on Frontiers of Computer Vision (IW-FCV2023) (2023.02.20 ~ 2023.02.23)

• Lamyanba Laishram, Md.Maklachur Rahman, Soon Ki Jung, Challenges and Applications of Face Deepfake, The 27th International Workshop on Frontiers of Computer Vision(IW-FCV 2021) (2021.02.22 ~ 2021.02.23)