Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search

Overview

The remarkable progress in text-to-video diffusion models enables the generation of photorealistic videos, although the content of these generated videos often includes unnatural movement or deformation, reverse playback, and motionless scenes. Recently, an alignment problem has attracted huge attention, where we steer the output of diffusion models based on some measure of the content's goodness. Because there is a large room for improvement of perceptual quality along the frame direction, we should address which metrics we should optimize and how we can optimize them in the video generation. In this paper, we propose diffusion latent beam search with lookahead estimator, which can select a better diffusion latent to maximize a given alignment reward at inference time. We then point out that improving perceptual video quality with respect to alignment to prompts requires reward calibration by weighting existing metrics. This is because when humans or vision language models evaluate outputs, many previous metrics to quantify the naturalness of video do not always correlate with the evaluation. We demonstrate that our method improves the perceptual quality evaluated on the calibrated reward, VLMs, and human assessment, without model parameter update, and outputs the best generation compared to greedy search and best-of-N sampling under much more efficient computational cost. The experiments highlight that our method is beneficial to many capable generative models, and provide a practical guideline: we should prioritize the inference-time compute allocation into enabling the lookahead estimator and increasing the search budget, rather than expanding the denoising steps.

How to Work

Trade-off between Generation Compute and Alignment

Qualitative Results for Inference-Time Text-to-Video Alignment

Comparison of text-to-video results between DLBS-LA, base models, and other sampling methods on SoTA models (Latte, CogVideoX, and Wan 2.1). DLBS-LA produces more dynamic, natural, and prompt-aligned videos than all baselines.

Prompt: Under a rainbow, a zebra kicks up a spray of water as it crosses a fast-flowing river.

Prompt: dog puts paws together

 Prompt: Two dogs chase each other, suddenly skidding around a sharp corner.

 Prompt: A person on a hoverboard colliding with a wall, the board stopping abruptly.

Human Evaluation

Reward Calibration for Preference Feedback

We then point out that the improvement of perceptual video quality, considering the alignment to prompts, requires reward calibration of existing metrics. When evaluating outputs using capable vision language models or human raters, many previous metrics for quantifying video naturalness do not always correlate with them. Optimal reward design for measuring perceptual quality highly depends on the degree of dynamics described in evaluation prompts. We design a weighted linear combination of multiple metrics, which is calibrated to perceptual quality and improves the correlation with VLM/human preference.

We select the best coefficients among brute-force candidates, based on the correlation with Gemini, for each set of prompts with a different dynamics grade. Prompts with a high dynamics grade, i.e., DEVIL-high, place greater weight on the dynamic degree. In contrast, prompts that describe slight motion, i.e., DEVIL-medium and DEVIL-static, place a smaller weight on it.

Qualitative Results for Calibrating Reward to Preference Feedback

We select the video with the highest reward out of 64 randomly generated candidates for each prompt, drawn from DEVIL-high, DEVIL-medium, DEVIL-static, and MSRVTT-test. 

Videos chosen using VLM-calibrated rewards achieve a more balanced quality compared to those relying on any single metric. 

DEVIL-high

Prompt: A storm sweeps an elephant into a raging river, carrying it away swiftly.

DEVIL-medium

Prompt: Macao beach with stone mountains aerial view from drone. travel destination. summer vacation. dominican republic

DEVIL-static

Prompt: black car is under the blue sign.

MSRVTT-test

Prompt: a yellow-haired girl is explaining about a game

Further Qualitative Results for Inference-Time Text-to-Video Alignment

 Wan 2.1-14B

 Prompt: A truck rushing away from a treacherous mountain pass during a blizzard, with sudden avalanches and rockslides adding to the danger.

 Prompt: The couple runs hand in hand to release a sky lantern, then watches it drift upward into the night sky, carried by the wind with the stars shining above.

 Prompt: A giraffe in a lifeguard outfit, sitting atop a high chair and watching over a crowded pool.

 Prompt: A sequence of a cow performing acrobatic stunts over a series of colorful, abstract platforms that morph shapes.

 Prompt: Thundering across a vast desert plain, the elephants race over dunes and dodge sandstorms, before swiftly traversing through a rocky canyon, bounding over boulders and leaping across narrow ravines.

 Prompt: A green monster made of plants walks through an airport.

 Prompt: A man in a suit fights monsters

 CogVideoX-5B

 Prompt: A giraffe navigating a city during a robot uprising, with quick cuts showing chaotic battles, explosions, and futuristic technology in a high-stakes escape scenario.

 Prompt: Aerial view shot of a cloaked figure elevating in the sky between skyscrapers.

 Prompt: a horse leading a wild stampede across a stormy beach with waves crashing, depicted with swift, sweeping camera moves, cinematic composition.

 Prompt: A futuristic robot uprising, ((lasers firing)), metallic drones, explosions, debris, ((screaming civilians)), dystopian cityscape.

 Prompt: A hot air balloon descending back to the ground.

 Prompt: A dog made of ice melts completely in a hot summer day

 Prompt: The cat tore across the living room, jumping over toys and furniture to catch the mouse.

 Prompt: Racing the sunset, a giraffe charges across the horizon, shadows stretching long.

 Prompt: futuristic sports cars racing on a vertical loop track against a sci-fi cityscape, cars defying gravity, ((speed trails)), (dizzying heights), (spectacular crashes), the thrill of cutting-edge technology.

 Latte

Prompt: A cool dj teddy bear with sunglasses on top of turntable with video static

Prompt: some fake horses are standing around in a game

Prompt: Amidst a thunderstorm, a lightning bolt strikes a bicycle, setting it ablaze with crackling energy and lighting up the dark, rainy street.

Prompt: a man holds up a stuffed bear.

 Prompt: Aerial view.  cute girl in the coat drive on country road on the bicycle

Prompt: cat manages to hang on to dangling object

Prompt: a man holds a very large stick

Prompt: Against the wind, a lone horse gallops, mane streaming behind.

Prompt: A car drives through a wall of fire in a daring escape.

Prompt: The ice cracks beneath their feet, making the sheep skid and slide, rushing to solid ground.

 Prompt: WWI biplanes in a dogfight with canvas wings ripping, dramatic cloud backdrop, ultra-detailed.

Prompt: Cowboys drive group of horses at farming enterprise.

Prompt: dog passes in and out of view

Prompt: when you can see the first view of the full bike

Prompt: a band is playing music and people are dancing

Prompt: a man inside of a car is using his finger to point

 Prompt: some images of motorcycles are being shown on tv

 Prompt: someone is browsing a set of games on their console