Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search

Overview

The remarkable progress in text-to-video diffusion models enables the generation of photorealistic videos, although the content of these generated videos often includes unnatural movement or deformation, reverse playback, and motionless scenes. Recently, an alignment problem has attracted huge attention, where we steer the output of diffusion models based on some measure of the content's goodness. Because there is a large room for improvement of perceptual quality along the frame direction, we should address which metrics we should optimize and how we can optimize them in the video generation. In this paper, we propose diffusion latent beam search with lookahead estimator, which can select a better diffusion latent to maximize a given alignment reward at inference time. We then point out that improving perceptual video quality with respect to alignment to prompts requires reward calibration by weighting existing metrics. This is because when humans or vision language models evaluate outputs, many previous metrics to quantify the naturalness of video do not always correlate with the evaluation. We demonstrate that our method improves the perceptual quality evaluated on the calibrated reward, VLMs, and human assessment, without model parameter update, and outputs the best generation compared to greedy search and best-of-N sampling under much more efficient computational cost. The experiments highlight that our method is beneficial to many capable generative models, and provide a practical guideline: we should prioritize the inference-time compute allocation into enabling the lookahead estimator and increasing the search budget, rather than expanding the denoising steps.

How to Work

Trade-off between Generation Compute and Alignment

Qualitative Results for Inference-Time Text-to-Video Alignment

Comparison of text-to-video results between DLBS-LA, base models, and other sampling methods on SoTA models (Latte, CogVideoX, and Wan 2.1). DLBS-LA produces more dynamic, natural, and prompt-aligned videos than all baselines.

Prompt: Under a rainbow, a zebra kicks up a spray of water as it crosses a fast-flowing river.

Prompt: dog puts paws together

 Prompt: Two dogs chase each other, suddenly skidding around a sharp corner.

 Prompt: A person on a hoverboard colliding with a wall, the board stopping abruptly.

• For more qualitative results for inference-time text-to-video alignment, please take a look at the Gallery.

Human Evaluation

Reward Calibration for Preference Feedback

We then point out that the improvement of perceptual video quality, considering the alignment to prompts, requires reward calibration of existing metrics. When evaluating outputs using capable vision language models or human raters, many previous metrics for quantifying video naturalness do not always correlate with them. Optimal reward design for measuring perceptual quality highly depends on the degree of dynamics described in evaluation prompts. We design a weighted linear combination of multiple metrics, which is calibrated to perceptual quality and improves the correlation with VLM/human preference.

We select the best coefficients among brute-force candidates, based on the correlation with Gemini, for each set of prompts with a different dynamics grade. Prompts with a high dynamics grade, i.e., DEVIL-high, place greater weight on the dynamic degree. In contrast, prompts that describe slight motion, i.e., DEVIL-medium and DEVIL-static, place a smaller weight on it.

Qualitative Results for Calibrating Reward to Preference Feedback

We select the video with the highest reward out of 64 randomly generated candidates for each prompt, drawn from DEVIL-high, DEVIL-medium, DEVIL-static, and MSRVTT-test. 

Videos chosen using VLM-calibrated rewards achieve a more balanced quality compared to those relying on any single metric. 

DEVIL-high

Prompt: A storm sweeps an elephant into a raging river, carrying it away swiftly.

DEVIL-medium

Prompt: Macao beach with stone mountains aerial view from drone. travel destination. summer vacation. dominican republic

DEVIL-static

Prompt: black car is under the blue sign.

MSRVTT-test

Prompt: a yellow-haired girl is explaining about a game