TL;DR.

EnerVerse is a framework for generating future spaces represented as multi-view videos for robotic manipulation tasks. It uses chunkwise autoregressive generation and a sparse memory mechanism to produce infinite sequences with explicit end-of-sequence (EoS) control. We further integrate it with 4D Gaussian Splatting (4DGS) to construct a data flywheel for the sim2real adaption. Plugged with a naive policy head, it achieves state-of-the-art performance in robotic manipulation benchmarks, demonstrating the effectiveness of its space generation prior.

Abstract: We introduce EnerVerse, a comprehensive framework for embodied future space generation specifically designed for robotic manipulation tasks.  EnerVerse seamlessly integrates convolutional and bidirectional attention mechanisms for inner-chunk space modeling, thereby ensuring low-level consistency and continuity. Recognizing the inherent redundancy in video data, we propose a sparse memory context in conjunction with a chunkwise unidirectional generative paradigm to facilitate the generation of infinitely long sequences. To further augment robotic capabilities, we introduce the Free Anchor View (FAV) space, which offers flexible perspectives that enhance observation and analysis. The FAV space mitigates motion modeling ambiguity, removes physical constraints in confined environments, and significantly improves the robot’s generalization and adaptability across a variety of tasks and settings. To address the prohibitive costs and labor intensity associated with acquiring multi-camera observations, we present a data engine pipeline that integrates a generative model with 4D Gaussian Splatting (4DGS). This pipeline capitalizes on the generative model's robust generalization capabilities and the spatial constraints provided by 4DGS, enabling an iterative enhancement of data quality and diversity, thus creating a data flywheel effect that effectively narrows the sim-to-real gap. Finally, our experiments demonstrate that the embodied future space generation prior substantially enhances policy predictive capabilities, resulting in improved overall performance, particularly in long-range robotic manipulation tasks.

Free Anchor View Definition

Visualization of FAVs generation on the LIBERO benchmark. Anchor View 1 represents the observation image captured by a mounted camera. Anchor View 2 and Anchor View 3 are generated by rendering from a point cloud reconstructed from Anchor View 1 using depth wrapping.

Real-World Data Flywheels with EnerVerse and 4DGS

Observation images captured from multiple cameras, along with rendered images from anchor views, are processed by the multi-view video generator to produce denoised multi-view videos. These videos, paired with their corresponding camera poses, are utilized in 4D Gaussian Splatting (4D GS) for 4D scene reconstruction. The reconstructed content is rendered from anchor views to generate high-precision images, which are iteratively fed back into the pipeline to enhance motion consistency and reconstruction quality. This iterative loop combines geometric consistency with generative refinement, delivering high-fidelity outputs for tasks such as robotic manipulation.

Multi-View Video Generation Results

Video Generation Results on LIBERO Benchmark

Note that all the three views are generated by our model, and the consistency across views highlights the geometric and spatial information learned by our model.

The instruction in () indicates the finished pre-steps

Video Generation Results with Real-World Data

Single-View Video Generation

Qualitative comparison with DynamiCrafter on RT-1

Since ours predict EOS frame at 42th frame for this task, we visualize 8th, 16th, 24th and 41th frame sampled from both generated sequence. The sequences generated by DynamiCrafter(FN) did not maintain the logic of the long-range task, producing many hallucinations as the sequence grew. In contrast, the sequence generated by \OURS was logically coherent, continuously and completely generating the future space of the entire task, and accurately predicting the EOS (End of Sequence) frame.

Policy Evaluation Results

In the policy prediction experiment, the action head adopts the Diffusion Policy (DP) architecture, with a total of 190M parameters. For the condition of the DP head, we utilize the feature before middle block of the UNet in the first denoise step, and calculate the mean value over spatial dimension

Our Policy Rollouts on LIBERO Benchmarks