Related Work

Style-based GANs build on progressive growing GANs [1, 2, 3, 4]. They implicitly learn hierarchical latent styles for image generation and manipulate the per-channel mean and variance to control the style of an image effectively. As shown in the figure below, StyleGAN takes style vectors (output by a mapping network) and stochastic variation (provided by the noise layers) as inputs for image synthesis. This offers control over the style of generated images at different levels of detail and makes possible the generation of photorealistic images. In this work, we use StyleGAN [1] as base model for our experiments.

Several methods [7, 8, 9] have been developed to solve the above equation with a formulation based on learning, optimization, or both. A learning-based inversion method aims to learn an encoder network to map an image into the latent space such that the reconstructed image based on the latent code looks as similar to the original one as possible. An optimization-based inversion approach directly solves the objective function through back-propagation to find a latent code that minimizes pixel-wise reconstruction loss. A hybrid approach first uses an encoder to generate initial latent code and then refines it with an optimization algorithm.

A key factor that affects the quality of GAN inversion is whether an inverted encoding lies within the region represented by training examples in the latent space. A few recent approaches have proposed to force the inverted code to lie within this latent region by employing constraints on the target image at pixel level, or directly in the latent space. A recent technique [6] enforces this constraint on the w latent space by training an auxiliary discriminator that learns to differentiate between real and fake w latent vectors (shown in the figure below). Our network design also takes inspiration from this work.

The network architecture in [6]. The FC block is from StyleGAN that maps latent codes z from a Gaussian distribution to the w latent space. Only green blocks are trainable.

In order to explore robustness of latent space, interpolation is commonly used [1, 2, 3, 4]. However, to the best of our knowledge, latent space robustness when viewed from an adversarial perspective has received very limited focus.

Fast Gradient Sign Method (FGSM) [10] and Projected Gradient Descent (PGD) [11] are adversarial attacks which perform gradient ascent with respect to the classification loss to arrive at perturbed images which fools the classifier. In this work, we modify PGD to work with other loss functions more suited for generative models.