Latent Space Robustness of Generative Models
Group 7: Rohan Agarwal, Anirudh S Chakravarthy, Shefali Srivastava
Motivation
Generative models such as StyleGAN have shown very promising results for various tasks such as novel image synthesis for faces and animals. However, while using such GANs for face generation, we often encounter cases of non-photorealistic generations. In some cases, the generated images look like faces but consist of certain artefacts. In other cases, the images look nothing like faces.
In this project, we aim to formally establish the existence of such failure modes in GANs. Furthermore, if we are successful and can identify these "holes" in the latent space, this will be very useful for industry applications which employ generative models. Concretely, strategies can be employed to prevent sampling from these holes to ensure that the users always encounter non-noisy and photorealistic images. Finally, we also hope to motivate research towards overcoming this problem of holes in the latent space.
Technical Focus
Despite the fact that these failure cases are well known in literature, little work has been done to study their causes. Previous research uses interpolation to show the robustness of the latent space, but we raise the question whether this is actually a representative result. In particular, some regions in the latent space may exhibit the desired robustness using interpolation, but this cannot be generalized across the entire latent space.
Generative models such as GANs do not model the data distribution of the latent space directly. Instead, this is inferred through random sampling from the original distribution. Often, this is a standard normal distribution. We hypothesize that regions exist in the latent space from which insufficient latent codes are sampled during training. Therefore, the generative model may exhibit erratic behaviour (i.e. generate junk outputs) for latent codes sampled from these regions. We term these region as holes.
Through our study, we aim to verify the existence of these holes and define a formal procedure to arrive at these latent holes. To this end, we use an adversarial attack over an appropriately defined loss function to land at these holes.
Our idea to arrive at latent holes using an adversarial attack. Given a randomly sampled latent vector (z), we generate the corresponding image using a generator (G). By performing gradient ascent with respect to a loss function over z, we hope to arrive at adversarial holes.