Problem Setting

Best Viewed at https://sites.google.com/andrew.cmu.edu/multigan-distillation (CMU access)

Code available here

Overview

Our main goal is to study knowledge transfer in GANs from various perspectives. We explore two broad scenarios:

Learning with limited target data, where we wish to learn a GAN on a given (limited) target dataset mitigating issues such as mode collapse.
Learning with no target data, where the aim is to fuse the distribution of a set of source GANs.

Learning with limited target data

In this setting, we have very few images (target data) to train a GAN. The goal is to leverage the prior knowledge captured by a pre-trained source GAN to achieve a better generalization of the target data.

- We first consider the case where a single source GAN is trained on a dataset having a different (but related) data distribution. Knowledge is transferred by fine-tuning the GAN onto the target dataset with a special fine-tuning paradigm which yields a better transfer than simple fine-tuning.
- Then, we extend this idea to multi-source GANs, a more practical but challenging scenario, wherein there is a distribution shift between each pair. The key idea is to selectively transfer the semantics most relevant to the target dataset. Again, here we employ a special fine-tuning strategy to transfer to the target dataset.

Explore the analysis here

Learning with no target data

In this scenario, we don't have access to a high-quality target dataset on which the GAN must be trained. Instead, the goal would be to fuse the distributions learned by the source GANs and learn a target GAN that can generate the combined data. As such, one can consider the collection of source GANs to implicitly capture the target distribution.

Similar to the previous scenario, we first consider distribution transfer from a single source GAN. The challenge here is to faithfully reconstruct the source distribution using a different target GAN model. The performance of this model would indicate an upper bound for what can be inherited using the knowledge transfer algorithm.
We extend this idea to the multi-source setting wherein we wish to fuse the data distribution of each GAN.

Explore the analysis here

References

[1] Hinton et al, “Distilling the Knowledge in a Neural Network”, NeurIPS Deep Learning and Representation Learning Workshop (2015).

[2] Karras et al, “Analyzing and Improving the Image Quality of StyleGAN”, arXiv:1912.04958 (2020).

[3] Addepalli et al, “DeGAN: Data-Enriching GAN for Retrieving Representative Samples”, AAAI (2020).

[4] Kurmi et al, “Domain Impression: A Source Data Free Domain Adaptation Method”, WACV (2021).

[5] Kundu et al, “Universal Source-Free Domain Adaptation”, CVPR (2020).

[6] Kundu et al, “Towards Inheritable Models for Open-Set Domain Adaptation”, CVPR (2020).

[7] Wang et al, “Adversarial Learning of Portable Student Networks”, AAAI (2018).

[8] Chen et al, “Distilling Portable Generative Adversarial Networks for Image Translation”, AAAI (2020).

[9] Wang et al, “KDGAN: Knowledge Distillation with Generative Adversarial Networks”, NeuRIPS (2018).

[10] Chang et al, “TinyGAN: Distilling BigGAN for Conditional Image Generation”, ACCV (2020).

[11] Aguinaldo et al, “Compressing GANs using Knowledge Distillation”, arXiv:1902.00159 (2019).

[12] Isola et al, “Image-to-Image Translation with Conditional Adversarial Nets”, CVPR (2017).

[13] Lin et al, “Anycost GANs for Interactive Image Synthesis and Editing”, CVPR (2021).

[14] Sankaranarayanan et al, “Generate To Adapt: Aligning Domains using Generative Adversarial Networks”, CVPR (2018).

[15] Li et al, “GAN Compression: Efficient Architectures for Interactive Conditional GANs”, CVPR (2020).

[16] Li et al, "Semantic relation preserving knowledge distillation for image-to-image translation", ECCV (2020).

[17] Lopes et al, “Data-Free Knowledge Distillation for Deep Neural Networks”, NeurIPS Workshop on Learning with Limited Data (2017).

[18] Wang et al, "MineGAN: effective knowledge transfer from GANs to target domains with few images", CVPR (2020)

Page updated

Report abuse