Wan-Cyuan Fan✦, Yen-Chun Chen✝, Dongdong Chen✝, Yu Cheng✝, Lu Yuan✝, Yu-Chiang Frank Wang✦★
✦National Taiwan University ✝Microsoft Corporation ★NVIDIA
r09942092@ntu.edu.tw frankwang@nvidia.com
{yen-chun.chen, dochen, yu.cheng, luyuan}@microsoft.com
AAAI 2023
Diffusion models (DMs) have shown great potential for high-quality image synthesis. However, when it comes to producing images with complex scenes, how to properly describe both image global structures and object details remains a challenging task. In this paper, we present Frido, a Feature Pyramid Diffusion model performing a multi-scale coarse-to-fine denoising process for image synthesis. Our model decomposes an input image into scale-dependent vector quantized features, followed by a coarse-to-fine modulation for producing image output. During the above multi-scale representation learning stage, additional input conditions like text, scene graph, or image layout can be further exploited. Thus, Frido can be also applied for conditional or cross-modality image synthesis. We conduct extensive experiments over various unconditioned and conditional image generation tasks, ranging from text-to-image synthesis, layout-to-image, scene-graph-to-image, to label-to-image. More specifically, we achieved state-of-the-art FID scores on five benchmarks, namely layout-to-image on COCO and OpenImages, scene-graph-to-image on COCO and Visual Genome, and label-to-image on COCO.
Figure 1: Illustration of Frido. Given a cross-modal con- dition, Frido generates images in 1) a coarse-to-fine man- ner from structure to object details, producing outputs with 2) high semantic correctness and quality. Note that existing models such as the LDMs are not designed to distinguish between high/low-level visual information.
An inlet filled with boats of all kinds.
A pile of orange sitting on a market shelf.
Plates loaded with thanksgiving dinner and dessert with two glasses.
A breakfast bowl consisting of blueberry sconce and strawberry with side of coffee
A man with blue shirt standing next to skis in snowy forest.
Stark white applications stand out against brown wooden cabinets.
People are standing on the corner of a street.
Subway car is either coming or going outside of a subway stop
A river is calm as a picturesque brick bridge sits in the background.
Two small giraffes stand in a grassy savanna.
A person that is in the grass with a kite.
A beach unbrella on a beach with two lawn chairs
This work is supported in part by the National Science and Technology Council of Taiwan under grant NSTC 111- 2634-F-002-020.
We thank the National Center for High-performance Computing (NCHC), Taiwan Comput- ing Cloud (TWCC), and Novatek Foundation for providing computational resources and extra funding.
We are immensely grateful to Andreas Blattmann, a co-author of LDM (Rombach et al. 2022), for sharing his valuable ex- perience in implementing diffusion models, which was very helpful in the early stage of this work.
We thank Cheng-Fu Yang and Chiao-An Yang, our colleagues in the Vision and Learning Lab at NTU, for their feedback that helps polish this paper.
@inproceedings{fan2022frido,
title={Frido: Feature pyramid diffusion for complex scene image synthesis},
author={Fan, Wan-Cyuan and Chen, Yen-Chun and Chen, DongDong and Cheng, Yu and Yuan, Lu and Wang, Yu-Chiang Frank},
booktitle={AAAI},
year={2023}
}