Embedded Files
Table of Contents
How to Work
How to Work
Diffusion latent beam search (DLBS) seeks a better diffusion path over the reverse process; sampling K latents per beam and possessing B beams for the next step, which mitigates the effect from inaccurate argmax.
Lookahead estimator notably reduces the noise at latent reward evaluation by interpolating the rest of the time steps from the current latent with deterministic DDIM.
Trade-off between Generation Compute and Alignment
Trade-off between Generation Compute and Alignment
DLBS achieves much better computational-efficiency than best-of-N (BoN), as achieving higher performance gains under the same execution time.
LA estimator (DLBS-LA) could remarkably boost efficiency only with marginal overhead on top of DLBS.
Human Evaluation
Human Evaluation
We perform pairwise comparisons between DLBS-LA and GS(KB=1).
The results confirm that, whatever models or prompts we choose, the quality of content generated by DLBS-LA consistently outperforms that of a baseline.
Qualitative Results for Calibrating Reward to Preference Feedback
Qualitative Results for Calibrating Reward to Preference Feedback
We select the video with the highest reward out of 64 randomly generated candidates for each prompt, drawn from DEVIL-high, DEVIL-medium, DEVIL-static, and MSRVTT-test.
Videos chosen using VLM-calibrated rewards achieve a more balanced quality compared to those relying on any single metric.
DEVIL-high
Prompt: A storm sweeps an elephant into a raging river, carrying it away swiftly.
A storm sweeps an elephant into a raging river, carrying it away swiftly._motion_smoothness.mp4Motion Smoothness (lack of motion)
A storm sweeps an elephant into a raging river, carrying it away swiftly._dynamic_degree.mp4Dynamic Degree (prompt misalignment)
A storm sweeps an elephant into a raging river, carrying it away swiftly._aesthetic_quality.mp4Aesthetic Quality (lack of motion)
A storm sweeps an elephant into a raging river, carrying it away swiftly._gemini_calibrated.mp4Gemini Calibrated Reward
DEVIL-medium
Prompt: Macao beach with stone mountains aerial view from drone. travel destination. summer vacation. dominican republic
Macao beach with stone mountains aerial view from drone. travel destination. summer vacation. dominican republic_subject_consistency.mp4Subject Consistency (lack of motion)
Macao beach with stone mountains aerial view from drone. travel destination. summer vacation. dominican republic_imaging_quality.mp4Imaging Quality (prompt misalignment)
Macao beach with stone mountains aerial view from drone. travel destination. summer vacation. dominican republic_tv_consistency.mp4Text-Video Consistency
Macao beach with stone mountains aerial view from drone. travel destination. summer vacation. dominican republic_gpt_calibrated.mp4GPT Calibrated Reward
DEVIL-static
Prompt: black car is under the blue sign.
black car is under the blue sign._motion_smoothness.mp4Motion Smoothness (lack of motion, prompt misalignment)
black car is under the blue sign._dynamic_degree.mp4Dynamic Degree (lack of consistency, prompt misalignment)
black car is under the blue sign._aesthetic_quality.mp4Aesthetic Quality (prompt misalignment)
black car is under the blue sign._gemini_calibrated.mp4Gemini Calibrated Reward
MSRVTT-test
Prompt: a yellow-haired girl is explaining about a game
a yellow-haired girl is explaining about a game_subject_consistency.mp4Subject Consistency / Motion Smoothness (lack of motion)
a yellow-haired girl is explaining about a game_dynamic_degree.mp4Dynamic Degree (lack of consistency)
a yellow-haired girl is explaining about a game_imaging_quality.mp4Imaging Quality (prompt misalignment)
a yellow-haired girl is explaining about a game_gpt_calibrated_reward.mp4GPT Calibrated Reward
Qualitative Results for Inference-Time Text-to-Video Alignment
Qualitative Results for Inference-Time Text-to-Video Alignment
We evaluate prompts from DEVIL-high, DEVIL-medium, DEVIL-static, and MSRVTT-test with Latte, serving as our base text-to-video model.
DLBS-LA generates more dynamic and prompt-aligned videos than GS or BoN.
DEVIL-high
Prompt: Under a rainbow, a zebra kicks up a spray of water as it crosses a fast-flowing river.
video_K1_B1.mp4GS (KB=1)
video_K32_B1.mp4GS (KB=32)
video_K8_B4.mp4DLBS (KB=32)
video_K4_B2_LA6.mp4DLBS-LA (KB=8, T'=6)
DEVIL-medium
Prompt: A cool dj teddy bear with sunglasses on top of turntable with video static
video_K1_B1.mp4GS (KB=1)
video_K32_B1.mp4GS (KB=32)
video_K4_B8.mp4DLBS (KB=32)
video_K4_B2_LA6.mp4DLBS-LA (KB=8, T'=6)
DEVIL-static
Prompt: dog puts paws together
video_K1_B1.mp4GS (KB=1)
video_K1_B32.mp4BoN (KB=32)
video_K16_B2.mp4DLBS (KB=32)
video_K4_B2_LA6.mp4DLBS-LA (KB=8, T'=6)
MSRVTT-test
Prompt: some fake horses are standing around in a game
video_K1_B1.mp4GS (KB=1)
video_K32_B1.mp4GS (KB=32)
video_K4_B8.mp4DLBS (KB=32)
video_K4_B2_LA6.mp4DLBS-LA (KB=8, T'=6)
Qualitative Results with recent SoTA models
Qualitative Results with recent SoTA models
We also show qualitative results for inference-time text-to-video alignment with three SoTA models: Wan 2.1-14B, CogVideoX-5B, and Latte.
Wan 2.1-14B
Wan 2.1-14B
Prompt: A truck rushing away from a treacherous mountain pass during a blizzard, with sudden avalanches and rockslides adding to the danger.
A truck rushing away from a treacherous mountain pass during a blizzard, with sudden avalanches and rockslides adding to the danger.-0000.mp4GS (KB=1)
A truck rushing away from a treacherous mountain pass during a blizzard, with sudden avalanches and rockslides adding to the danger.-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: A person on a hoverboard colliding with a wall, the board stopping abruptly.
A person on a hoverboard colliding with a wall, the board stopping abruptly.-0000.mp4GS (KB=1)
A person on a hoverboard colliding with a wall, the board stopping abruptly.-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: The couple runs hand in hand to release a sky lantern, then watches it drift upward into the night sky, carried by the wind with the stars shining above.
The couple runs hand in hand to release a sky lantern, then watches it drift upward into the night sky, carried by the wind with the stars shining above.-0000.mp4GS (KB=1)
The couple runs hand in hand to release a sky lantern, then watches it drift upward into the night sky, carried by the wind with the stars shining above.-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: A giraffe in a lifeguard outfit, sitting atop a high chair and watching over a crowded pool.
A giraffe in a lifeguard outfit, sitting atop a high chair and watching over a crowded pool.-0000.mp4GS (KB=1)
A giraffe in a lifeguard outfit, sitting atop a high chair and watching over a crowded pool.-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: A sequence of a cow performing acrobatic stunts over a series of colorful, abstract platforms that morph shapes.
A sequence of a cow performing acrobatic stunts over a series of colorful, abstract platforms that morph shapes.-0000.mp4GS (KB=1)
A sequence of a cow performing acrobatic stunts over a series of colorful, abstract platforms that morph shapes.-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: Thundering across a vast desert plain, the elephants race over dunes and dodge sandstorms, before swiftly traversing through a rocky canyon, bounding over boulders and leaping across narrow ravines.
Thundering across a vast desert plain, the elephants race over dunes and dodge sandstorms, before swiftly traversing through a rocky canyon, bounding over boulders and leaping across narrow ravines.-0000.mp4GS (KB=1)
Thundering across a vast desert plain, the elephants race over dunes and dodge sandstorms, before swiftly traversing through a rocky canyon, bounding over boulders and leaping across narrow ravines.-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: A green monster made of plants walks through an airport.
A green monster made of plants walks through an airport.-0000.mp4GS (KB=1)
A green monster made of plants walks through an airport.-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: A man in a suit fights monsters
A man in a suit fights monsters -0000.mp4GS (KB=1)
A man in a suit fights monsters -0000.mp4DLBS-LA (KB=8, T'=6)
CogVideoX-5B
CogVideoX-5B
Prompt: Two dogs chase each other, suddenly skidding around a sharp corner.
Two dogs chase each other, suddenly skidding around a sharp corner.-0000.mp4GS (KB=1)
Two dogs chase each other, suddenly skidding around a sharp corner.-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: A giraffe navigating a city during a robot uprising, with quick cuts showing chaotic battles, explosions, and futuristic technology in a high-stakes escape scenario.
A giraffe navigating a city during a robot uprising, with quick cuts showing chaotic battles, explosions, and futuristic technology in a high-stakes escape scenario.-0000.mp4GS (KB=1)
A giraffe navigating a city during a robot uprising, with quick cuts showing chaotic battles, explosions, and futuristic technology in a high-stakes escape scenario.-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: Aerial view shot of a cloaked figure elevating in the sky between skyscrapers.
Aerial view shot of a cloaked figure elevating in the sky between skyscrapers.-0000.mp4GS (KB=1)
Aerial view shot of a cloaked figure elevating in the sky between skyscrapers.-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: a horse leading a wild stampede across a stormy beach with waves crashing, depicted with swift, sweeping camera moves, cinematic composition.
a horse leading a wild stampede across a stormy beach with waves crashing, depicted with swift, sweeping camera moves, cinematic composition.-0000.mp4GS (KB=1)
a horse leading a wild stampede across a stormy beach with waves crashing, depicted with swift, sweeping camera moves, cinematic composition.-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: A futuristic robot uprising, ((lasers firing)), metallic drones, explosions, debris, ((screaming civilians)), dystopian cityscape.
A futuristic robot uprising, ((lasers firing)), metallic drones, explosions, debris, ((screaming civilians)), dystopian cityscape.-0000.mp4GS (KB=1)
A futuristic robot uprising, ((lasers firing)), metallic drones, explosions, debris, ((screaming civilians)), dystopian cityscape.-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: A hot air balloon descending back to the ground.
A hot air balloon descending back to the ground.-0000.mp4GS (KB=1)
A hot air balloon descending back to the ground.-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: A dog made of ice melts completely in a hot summer day
A dog made of ice melts completely in a hot summer day-0000.mp4GS (KB=1)
A dog made of ice melts completely in a hot summer day-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: The cat tore across the living room, jumping over toys and furniture to catch the mouse.
The cat tore across the living room, jumping over toys and furniture to catch the mouse.-0000.mp4GS (KB=1)
The cat tore across the living room, jumping over toys and furniture to catch the mouse.-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: Racing the sunset, a giraffe charges across the horizon, shadows stretching long.
Racing the sunset, a giraffe charges across the horizon, shadows stretching long.-0000.mp4GS (KB=1)
Racing the sunset, a giraffe charges across the horizon, shadows stretching long.-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: futuristic sports cars racing on a vertical loop track against a sci-fi cityscape, cars defying gravity, ((speed trails)), (dizzying heights), (spectacular crashes), the thrill of cutting-edge technology.
futuristic sports cars racing on a vertical loop track against a sci-fi cityscape, cars defying gravity, ((speed trails)), (dizzying heights), (spectacular crashes), the thrill of cutting-edge technology.-0000.mp4GS (KB=1)
futuristic sports cars racing on a vertical loop track against a sci-fi cityscape, cars defying gravity, ((speed trails)), (dizzying heights), (spectacular crashes), the thrill of cutting-edge technology.-0000.mp4DLBS-LA (KB=8, T'=6)
Latte
Latte
Prompt: Amidst a thunderstorm, a lightning bolt strikes a bicycle, setting it ablaze with crackling energy and lighting up the dark, rainy street.
Amidst a thunderstorm, a lightning bolt strikes a bicycle, setting it ablaze with crackling energy and lighting up the dark, rainy street.-0000.mp4GS (KB=1)
Amidst a thunderstorm, a lightning bolt strikes a bicycle, setting it ablaze with crackling energy and lighting up the dark, rainy street.-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: a man holds up a stuffed bear.
a man holds up a stuffed bear.-0000.mp4GS (KB=1)
a man holds up a stuffed bear.-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: Aerial view. cute girl in the coat drive on country road on the bicycle
Aerial view. cute girl in the coat drive on country road on the bicycle-0000.mp4GS (KB=1)
Aerial view. cute girl in the coat drive on country road on the bicycle-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: cat manages to hang on to dangling object
cat manages to hang on to dangling object-0000.mp4GS (KB=1)
cat manages to hang on to dangling object-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: a man holds a very large stick
a man holds a very large stick-0000.mp4GS (KB=1)
a man holds a very large stick-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: Against the wind, a lone horse gallops, mane streaming behind.
Against the wind, a lone horse gallops, mane streaming behind.-0000.mp4GS (KB=1)
Against the wind, a lone horse gallops, mane streaming behind.-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: A car drives through a wall of fire in a daring escape.
A car drives through a wall of fire in a daring escape.-0000.mp4GS (KB=1)
A car drives through a wall of fire in a daring escape.-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: The ice cracks beneath their feet, making the sheep skid and slide, rushing to solid ground.
The ice cracks beneath their feet, making the sheep skid and slide, rushing to solid ground.-0000.mp4GS (KB=1)
The ice cracks beneath their feet, making the sheep skid and slide, rushing to solid ground.-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: WWI biplanes in a dogfight with canvas wings ripping, dramatic cloud backdrop, ultra-detailed.
WWI biplanes in a dogfight with canvas wings ripping, dramatic cloud backdrop, ultra-detailed.-0000.mp4GS (KB=1)
WWI biplanes in a dogfight with canvas wings ripping, dramatic cloud backdrop, ultra-detailed.-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: Cowboys drive group of horses at farming enterprise.
Cowboys drive group of horses at farming enterprise.-0000.mp4GS (KB=1)
Cowboys drive group of horses at farming enterprise.-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: dog passes in and out of view
dog passes in and out of view-0000.mp4GS (KB=1)
dog passes in and out of view-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: when you can see the first view of the full bike
when you can see the first view of the full bike-0000.mp4GS (KB=1)
when you can see the first view of the full bike-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: a band is playing music and people are dancing
a band is playing music and people are dancing-0000.mp4GS (KB=1)
a band is playing music and people are dancing-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: a man inside of a car is using his finger to point
a man inside of a car is using his finger to point-0000.mp4GS (KB=1)
a man inside of a car is using his finger to point-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: some images of motorcycles are being shown on tv
some images of motorcycles are being shown on tv-0000.mp4GS (KB=1)
some images of motorcycles are being shown on tv-0000.mp4DLBS-LA (KB=8, T'=6)
Prompt: someone is browsing a set of games on their console
someone is browsing a set of games on their console-0000.mp4GS (KB=1)
someone is browsing a set of games on their console-0000.mp4DLBS-LA (KB=8, T'=6)
Page updated
Google Sites
Report abuse