Cartoon-to-Drama Image Style Transfer
📌 Insight from the project
Research on GAN-based Image Transfer
Conducted research using GAN models like CycleGAN and StyleGAN to transform webtoon images into drama images and vice versa.
Applied various techniques to improve the quality of generated images by performing unpaired image-to-image translation between different domains.
Challenges in Generating Human Faces
Unlike drawings, human faces are difficult to recognize when the facial structure proportions are distorted.
When generating real human faces, it is challenging to represent the depth within the face.
Introduction
Research Background
Adaptation of Webtoons into Visual Media
Webtoons are increasingly being adapted into visual media such as dramas and films.
Webtoons and dramas have distinct visual styles, and transforming these styles seamlessly to provide viewers with a new visual experience is a significant challenge.
Research Goal
Develop a GAN-based model that can maintain visual consistency while naturally transforming images between the domains of webtoons and dramas.
Utilize models such as CycleGAN and StyleGAN to transform facial features between webtoon and drama characters.
Dataset
Data Collection
Toon Image
Collected images from the webtoon on Naver Webtoon.
Crawled the faces of the main characters to build the dataset.
Drama Image
Extracted images from YouTube videos of the drama adaptation of the webtoon.
Extracted images frame by frame, securing a total of 8,498 images.
Data Processing
Detected faces from both webtoon and drama images.
Removed backgrounds and classified the data by character.
Performed preprocessing while maintaining image consistency by considering face direction, speech bubbles, and backgrounds.
Converted both webtoon and drama images into sketches.
Model
CycleGAN-based Image Translation
Model Structure:
Performed image translation between webtoon and drama images using the CycleGAN model.
Result
The translation is not clean, likely due to the small amount of data.
The images generally turned out darker.
Sketch-based Image Translation
Model Structure:
Converted webtoon images into sketches.
Translated webtoon sketches into real human image sketches.
Generated photos from human sketches.
Image-to-Sketch
Image Sketching: Adopted the model structure from "Bridging Unpaired Facial Photos and Sketches by Line-drawings."
Sketch to Drama Conversion: Converted drama sketches into realistic drama images.
Webtoon sketch example
Drama sketch example
Sketch-to-Image
Sketch-to-Webtoon example
Sketch-to-Drama example
Colorization-based Image Translation
Model Structure:
Utilized a Reference-based Colorization model.
Discussion
Data Insufficiency: 1:1 transfer learning is difficult due to the lack of matching data between webtoon and drama.
Domain Shift: Significant style differences between webtoon and drama make image translation challenging.
Image Complexity: Webtoon images often depict various objects, making focusing solely on the characters difficult.
⇒ Need for an Additional Domain Transfer Model
: An additional domain shift model is needed to reduce the domain differences between webtoon and drama.
⇒ Building a Webtoon-Drama Pair Dataset
: To proceed similarly to general style transfer research, a paired dataset of both domains is required.