Due to the difficulty of acquiring extensive real-world data, robot simulation has become crucial for parallel training and sim-to-real transfer, highlighting the importance of scalable simulated robotic tasks. Foundation models have demonstrated impressive capacities in generative simulations to generate feasible robotic tasks autonomously. Nevertheless, this new paradigm underscores the challenge of adequately evaluating these autonomously generated tasks.  To address this, we propose a comprehensive evaluation framework tailored to generative simulations. 

Our framework segments evaluation into three core components: task generation quality, task diversity and generalizability. Our framework segments evaluation into three core aspects: quality, diversity, and generalization. For task quality, we evaluate the realism of the generated task as well as the completeness of the generated trajectories with large language models and vision-language models. In terms of diversity, we measure both task and data diversity through text similarity of task descriptions and world model loss trained on collected task trajectories respectively. For task-level generalization, we assess the zero-shot generalization ability on unseen tasks of a policy trained with multiple generated tasks.

Experiments conducted on three representative task generation pipelines demonstrate that the results from our framework are highly consistent with human evaluations, confirming the feasibility and validity of our approach. The findings reveal that while metrics of quality and diversity can be achieved through certain methods, no single approach excels across all metrics, suggesting a need for greater focus on balancing these different metrics. Additionally, our analysis further highlights the common challenge of low generalization capability faced by current works.

We leverage LLMs and VLMs to derive scene alignment score, which measures the consistency between the rendered scene and real-world task, and task completion score, referring to whether a task is completed  overcoming the limits of traditional hard-coded methods. 

To validate the efficacy of our method, we examine the consistency of our results with human evaluations for ten released tasks from RoboGen and GenSim. The score is calculated though dividing the Pearson correlation coefficient by the mean absolute error (MAE), where higher values indicate greater similarity.

As for the former, we calculate the similarity between embeddings of task descriptions. For the latter, we train a latent dynamics model following Dreamerv3((Hafner et al., 2024) on trajectories collected from the generated tasks and compute the prediction errors. Higher prediction errors imply more unfamiliar dynamics being experienced, indicating the diversity of the generated trajectories.