Learning World Models for Interactive Video Generation
Taiye Chen*¹ Xun Hu*² Zihan Ding*³ Chi Jin³
yeyutaihan@stu.pku.edu.cn xh4421@princeton.edu zihand@princeton.edu chij@princeton.edu
¹Peking University, ²University of Oxford, ³Princeton University
* Equal Contribution
NeurIPS 2025
[Paper] [Code]
Foundational world models must be both interactive and preserve spatiotemporal coherence for effective future planning with action choices. However, present models for long video generation have limited inherent world modeling capabilities due to two main challenges: compounding errors and insufficient memory mechanisms.
We enhance image-to-video models with interactive capabilities through additional action conditioning and autoregressive framework, and reveal that compounding error is inherently irreducible in autoregressive video generation, while insufficient memory mechanism leads to incoherence of world models. We propose video retrieval augmented generation (VRAG) with explicit global state conditioning, which significantly reduces long-term compounding errors and increases spatiotemporal consistency of world models.
In contrast, naive autoregressive generation with extended context windows and retrieval-augmented generation prove less effective for video generation, primarily due to the limited in-context learning capabilities of current video models. Our work illuminates the fundamental challenges in video world models and establishes a comprehensive benchmark for improving video generation models with internal world modeling capabilities.
A video world model should have capabilities to:
possess proprioception, i.e., be aware of its current state or position;
memorize what has been seen before;
recall from historical memory for future prediction with spatiotemporal coherence
therefore enables faithful long-term future prediction by maintaining awareness of its environment and generating predictions based on the current state and actions.
Overview of our VRAG framework for interactive video generation. The framework incorporates global state conditioning (position and orientation of the agent) and memory retrieval mechanisms (based on state or frame similarity) to ensure spatiotemporal consistency and mitigate error accumulation. During both training and inference, retrieved memory serves as context for joint self-attention in spatiotemporal DiT blocks. The model employs per-frame noise injection during training to facilitate autoregressive sampling at inference time.
Visual comparison of VRAG with ground truth videos on world coherence evaluation: top row is VRAG predicted videos, bottom row (with red blocks) is ground truth.
With 100 initial frames as history buffer, VRAG predicts 200 subsequent frames given the action sequences.
More evaluations are conducted over two dimensions: world coherence and compounding error.
DF 10
DF 20
YaRN
Frame Pack
History Buffer
Neural Memory
VRAG (Ours)
Ground Truth
We investigate the spatiotemporal consistency of internal world models by evaluating the predicted videos given initial frames and action sequences, comparing VRAG against Diffusion Forcing with different window sizes (DF10, DF20), YaRN for context extension, Frame Pack for context frame compression, History Buffer for heuristic-based context frames, Neural Memory with infini-attention, and ground truth. The visual results compare the predicted videos with 300 frames, and VRAG show superior performances in world coherence over all baselines.
SSIM scores (higher is better) are compared in above figure. Our experimental results demonstrate that expanding the window size from 10 to 20 frames in the baseline DF model improves world coherence, indicating that longer context windows enhance consistency. However, further context extension using YaRN shows no improvement over the vanilla DF model. Similarly, the History Buffer method fails to effectively utilize historical frames for spatiotemporal consistency without explicit in-context training. The Neural Memory method performs poorly due to its instability in model training, while our VRAG method achieves the best performance across all metrics, demonstrating its superior ability to maintain world coherence in generated videos.
df20_random2.mp4Diffusion Forcing with window size 20
rag_random2.mp4VRAG
We evaluate the compounding error in long video generation for consecutive 1200 frames. VRAG method generates higher-quality frames over long sequences, while other methods exhibit noticeable artifacts and inconsistencies.
DFoT
VRAG
GT
We conduct additional experiments in real-world setting beyond Minecraft simulation dataset to show generalization of our approach. Specifically, following the experimental setup of Diffusion Forcing Transformer (DFoT), our VRAG model is initialized from pre-trained DFoT and finetuned on the RealEstate10K dataset with additionally retrieved historical context. After fine-tuning for just 2 epochs (10% of the original training steps), our method significantly outperforms the DFoT baseline in terms of memorization capability.
@article{chen2025learning,
title={Learning World Models for Interactive Video Generation},
author={Chen, Taiye and Hu, Xun and Ding, Zihan and Jin, Chi},
journal={arXiv preprint arXiv:2505.21996},
year={2025}
}