Neural style transfer (NST) refers to the task of altering the style of an image from one domain to another. Originally the task of NST was performed on VGG-19 architecture. NST has immense use cases in the field of data augmentation, augmented reality, and photo editing and therefore has been considered one of the most critical problems in computer vision. Recent research has seen a rise in the usage of Generative Adversarial Networks (GANs) for the case of image translations. In this project, we explored the task of Neural Style Transfer using a Generative Adversarial Network. In particular, we plan to implement a novel GAN architecture that uses conditional GANs as its backbone and accepts as input an image (on which style transfer is to be applied) and random noise. We compared the results generated by these models to support our hypothesis.

• A ConvNet requires a large Dataset to process and train the neural network.

• Image synthesis procedure could take up to an hour on a Nvidia K40 GPU (depending on the exact image size and the stopping criteria for the gradient descent) [1]

• GANs currently generate the sharpest images.

• GANs work best in case of insufficient dataset.

• We plan to use multiple GANs such as cGAN and WCGAN to understand their performance on the task of neural style transfer.

• Receives style-image, a content-image and an input image (empty white image or a copy of a content-image.)

• Generate an image that has the minimal distance between both content and style images.

• Define a content loss (MSE of the feature activations between content and target image)

• And a style loss (MSE of the Gram matrices of the style image and the target image)

• Discriminator network: try to distinguish between real and fake images.

• Generator network: try to fool the discriminator by generating real-looking images.

The difference between the GAN and the cGAN is that cGAN is conditional. Meaning it takes an extra input that is the label along with the noise. This teaches the Generator to create images for a particular class, hence it is called conditioning.

• Discriminator Loss: D(x, y) - D(x, G(x, z))

• The discriminator tries to maximize this function

• Generator Loss: D(G(x, z))

• The generator tries to maximize this function

• Pre-trained CNN is used to extract the content and style of a given reference image.

• The extracted content and style are then used to train a cGAN.

• Training the generator network to produce synthesized images that match the style of the reference image.

• Training the discriminator network to distinguish between real and fake images.

• Feed the content of the input image and the style of the reference image into the generator network, which produces a synthesized image that combines the content and style of the input images.

• To train a condition GAN we created image pair from both the data sets.

• Pairing them was challenging.

• Sample input to cGAN for training.

• We trained two different flavors of condition GANs.

  1. General Adversarial loss + l1 loss

  2. Wasserstein Adversarial loss + l1 loss

• Due to limited computation resources we trained on ~400 sample.

• Training epochs = 100

• L1 weight = 100

• Approximate training time : 8hr (on Personal Macbook)

• Inference time : 15ms (image size : 256x256)

• We need to train again for different style images.

• Despite computation limitation the proposed GAN model achieve comparable results.

• GAN approach needs far less images compared to CNN based approach.

• CNN based approach can have very high inference time compared to GAN based approach.

• WCGAN trains much faster compared to cGAN producing comparable result.

• Changing the style image require re-training.

• Requires pair of training images.

• CNN based approach trained over low no of images looses the original image structures

• Some for complex GAN architecture like cycle GAN should be explored.

• cycle-GAN based approach avoided retraining when we change style image.

• To improve to image quality we can have different models for different kinds of data sets.

• With high computational resources we can produce high quality images.

• [1] L. A. Gatys, A. S. Ecker, and M. Bethge. A neural algorithm of artistic style. arXiv preprint arXiv:1508.06576

• [2] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020.

• [3] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017.

• [4] Y. Jing, Y. Yang, Z. Feng, J. Ye, Y. Yu, and M. Song. Neural style transfer: A review. IEEE transactions on visualization and computer graphics, 26(11):3365–3385, 2019.

• [5] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019.

• [6] S. Yang, J. Liu, Z. Lian, and Z. Guo. Awesome typography: Statistics-based text effects transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7464–7473, 2017.

• [7] L. Zhang, Y. Ji, X. Lin, and C. Liu. Style transfer for anime-sketches with enhanced residual u-net and auxiliary classifier gan. In 2017 4th IAPR Asian conference on pattern recognition (ACPR), pages 506–511. IEEE, 2017.

• [8] https://machinelearningmastery.com/how-to-develop-a-pix2pix-gan-for-image-to-image-translation/