We introduce Diffusion-based Audio Captioning (DAC), a non-autoregressive diffusion model tailored for diverse and efficient audio captioning. Although existing captioning models relying on language backbones have achieved remarkable success in various captioning tasks, their insufficient performance in terms of generation speed and diversity hinders the generation of large-scale and diverse text-audio pair data.
Our diffusion-based framework offers unique advantages stemming from its inherent stochasticity and holistic context modeling in captioning. Through rigorous evaluation, we demonstrate that DAC not only achieves SOTA performance levels compared to existing benchmarks in the caption quality, but also significantly outperforms them in terms of generation speed and diversity. To the best of our knowledge, DAC is the first model to leverage a diffusion-based framework within the audio captioning task. The success of DAC illustrates that text generation can also be seamlessly integrated with audio and visual generation tasks using a diffusion backbone, paving the way for a unified, audio-related generative model across different modalities.
YVjSEIRnLAh8
water is clanging, and a woman speaks.
YfsBR7e_X_0Y
a child speaks followed by a clicking.
Y_xylo5_IiaM
a baby cries and a woman speaks.
YCeRoaEcqUgM
wind blows loudly as waves crash by.
YCefFMA3klxk
a truck passes by and a horn blows.
YcPiSd5nJLrI
a woman screams, a loud roar and a loud explosion and a explosion.
Yy93cZqNCtks
loud gunshots followed by a male voice yelling and yelling and loud gunshot.
YxBZnvfniA1c
a man speaks followed by a child and a child laughing
Y7cHRSfbp7tc
a horse is trng while people talk in the background.
YQHfyKaOHSz4
a frog croaks and a buzzing tone and a man singing followed by music.
...