Jie Zhu, Yixiong Chen, Mingyu Ding, Ping Luo, Leye Wang, Jingdong Wang
Peking University, Johns Hopkins University, UC Berkeley, The University of Hong Kong, Baidu
NeurIPS 2024
MoLE
Text-to-image diffusion has attracted vast attention due to its impressive image-generation capabilities. However, when it comes to human-centric text-to-image generation, particularly in the context of faces and hands, the results often fall short of naturalness due to insufficient training priors. We alleviate the issue in this work from two perspectives. 1) From the data aspect, we carefully collect a human-centric dataset comprising approximately one million high-quality human-in-the-scene images and two specific sets of close-up images of faces and hands. These datasets collectively provide a rich prior knowledge base to enhance the human-centric image generation capabilities of the diffusion model. 2) On the methodological front, we propose a simple yet effective method called Mixture of Low-rank Experts (MoLE) by considering low-rank modules trained on close- up hand and face images respectively as experts. This concept draws inspiration from our observation of low-rank refinement, where a low-rank module trained by a customized close-up dataset has the potential to enhance the corresponding image part when applied at an appropriate scale. To validate the superiority of MoLE in the context of human-centric image generation compared to state-of-the-art, we construct two benchmarks and perform evaluations with diverse metrics and human studies.
Motivation of Our Method:
A low-rank refinement phenomenon that that low-rank module trained by a customized close-up dataset has potential to refine corresponding part under a proper scale weight.
Prompt: A man listening to music
Prompt: A girl eating an apple Prompt: A woman drinking milk
Mixture of Low-rank Experts (MoLE):
By regarding low-rank modules trained on customized datasets as experts, we formulate two experts in MoE form and combine with base model in a manner of soft miture assignment.
The framework of our method MoLE
We compare the generated images with four different models including SD v1.5, SD XL, VQ-Diffusion, and Versatile Diffusion. Pay more attention to the face and hand of the generated person.
More generated results of our MoLE. Pay more attention to the face and hand of generated person.
Please see our paper for more results.
