Wan-Cyuan Fan✦, Cheng-Fu Yang☀, Chiao-An Yang✝, Yu-Chiang Frank Wang✦★
✦National Taiwan University ☀UCLA ✝Purdue University ★NVIDIA
r09942092@ntu.edu.tw cfyang@cs.ucla.edu yang2300@purdue.edu frankwang@nvidia.com
AAAI 2023
We tackle the problem of target-free text-guided image manipulation, which requires one to modify the input reference image based on the given text instruction, while no ground truth target image is observed during training. To address this challenging task, we propose a Cyclic-Manipulation GAN (cManiGAN) in this paper, which is able to realize where and how to edit the image regions of interest. Specifically, the image editor in cManiGAN learns to identify and complete the input image, while cross-modal interpreter and reasoner are deployed to verify the semantic correctness of the output image based on the input instruction. While the former utilizes factual/counterfactual description learning for authenticating the image semantics, the latter predicts the "undo" instruction and provides pixel-level supervision for the training of cManiGAN. With such operational cycle-consistency, our cManiGAN can be trained in the above weakly supervised setting. We conduct extensive experiments on the datasets of CLEVR and COCO, and the effectiveness and generalizability of our proposed method can be successfully verified.
In this paper, we propose a cyclic-Manipulation GAN (cManiGAN) for target-free image manipulation. Due to the absence of ground truth target image during training, it is extremely challenging to identify where and how to edit the input reference image, so that the output would be semantically correct. To tackle the above two obstacles, our cManiGAN exploits self-supervised learning for enforcing the semantic correctness, while pixel-level guidance can be automatically observed. More specifically, the image editor of cManiGAN learns to locate/edit the image region of interest with global and local semantics observed. And, the modules of cross-modality interpreter and reasoner are deployed in cManiGAN. The former is introduced to verify the image semantics via factual/counter-factual description learning, while the latter infers the “undo” instruction and offers pixel-level supervision. As detailed later, the above design uniquely utilizes cross-modal cycle consistency and allows the weakly-supervised training our of cManiGAN.
This work is supported in part by the National Science and Technology Council of Taiwan under grant NSTC 111- 2634-F-002-020.
We thank the National Center for High-performance Computing (NCHC) and Taiwan Computing Cloud (TWCC) for providing computational and storage resources.
We are especially grateful to Dr. Yunhsuan Sung and Dr. Da-Cheng Juan from Google Research for giving their valuable research advice.