Cartoon-to-Drama Image Style Transfer

📌 Insight from the project

Research on GAN-based Image Transfer
- Conducted research using GAN models like CycleGAN and StyleGAN to transform webtoon images into drama images and vice versa.
- Applied various techniques to improve the quality of generated images by performing unpaired image-to-image translation between different domains.

Challenges in Generating Human Faces
- Unlike drawings, human faces are difficult to recognize when the facial structure proportions are distorted.
- When generating real human faces, it is challenging to represent the depth within the face.

Introduction

Research Background

Adaptation of Webtoons into Visual Media

Webtoons are increasingly being adapted into visual media such as dramas and films.
Webtoons and dramas have distinct visual styles, and transforming these styles seamlessly to provide viewers with a new visual experience is a significant challenge.

Research Goal

Develop a GAN-based model that can maintain visual consistency while naturally transforming images between the domains of webtoons and dramas.
Utilize models such as CycleGAN and StyleGAN to transform facial features between webtoon and drama characters.

Dataset

Data Collection

Toon Image
- Collected images from the webtoon on Naver Webtoon.
- Crawled the faces of the main characters to build the dataset.
Drama Image
- Extracted images from YouTube videos of the drama adaptation of the webtoon.
- Extracted images frame by frame, securing a total of 8,498 images.

Data Processing

Detected faces from both webtoon and drama images.
Removed backgrounds and classified the data by character.
Performed preprocessing while maintaining image consistency by considering face direction, speech bubbles, and backgrounds.
Converted both webtoon and drama images into sketches.

Model

CycleGAN-based Image Translation

Model Structure:
- Performed image translation between webtoon and drama images using the CycleGAN model.

Result
- The translation is not clean, likely due to the small amount of data.
- The images generally turned out darker.

Sketch-based Image Translation

Model Structure:
- Converted webtoon images into sketches.
- Translated webtoon sketches into real human image sketches.
- Generated photos from human sketches.

Image-to-Sketch
- Image Sketching: Adopted the model structure from "Bridging Unpaired Facial Photos and Sketches by Line-drawings."
- Sketch to Drama Conversion: Converted drama sketches into realistic drama images.

Webtoon sketch example

Drama sketch example

Sketch-to-Image

Sketch-to-Webtoon example

Sketch-to-Drama example

Colorization-based Image Translation

Model Structure:
- Utilized a Reference-based Colorization model.

Discussion

Data Insufficiency: 1:1 transfer learning is difficult due to the lack of matching data between webtoon and drama.
Domain Shift: Significant style differences between webtoon and drama make image translation challenging.
Image Complexity: Webtoon images often depict various objects, making focusing solely on the characters difficult.

⇒ Need for an Additional Domain Transfer Model

: An additional domain shift model is needed to reduce the domain differences between webtoon and drama.

⇒ Building a Webtoon-Drama Pair Dataset

: To proceed similarly to general style transfer research, a paired dataset of both domains is required.

Page updated

Google Sites

Report abuse