ReGen: Generative Robot Simulation via Inverse Design

Simulation plays a key role in scaling robot learning and validating policies, but constructing simulations remains labor-intensive. In this paper, we introduce \ourmethod{}, a generative simulation framework that automates this process using inverse design. Given an agent's behavior (such as a motion trajectory or objective function) and its textual description, we infer the underlying scenarios and environments that could have caused the behavior.

Our approach leverages large language models to construct and expand a graph that captures cause-and-effect relationships and relevant entities with properties in the environment, which is then processed to configure a robot simulation environment. Our approach supports (i) augmenting simulations based on ego-agent behaviors, (ii) controllable, counterfactual scenario generation, (iii) reasoning about agent cognition and mental states, and (iv) reasoning with distinct sensing modalities, such as braking due to faulty GPS signals.

We demonstrate our method in autonomous driving and robot manipulation tasks, generating more diverse, complex simulated environments compared to existing simulations with high success rates, and enabling controllable generation for corner cases. This approach enhances the validation of robot policies and supports data or simulation augmentation, advancing scalable robot learning for improved generalization and robustness.

Gallery

ego_template_1_scenario_1.mp4

Crossing with a malfunctioned traffic light

ego_template_1_scenario_2.mp4

Advancing after distracted driver moves

ego_template_1_scenario_3.mp4

Waiting for intersection to clear

ego_template_1_scenario_5.mp4

Yielding for ambulance at intersection

ego_template_1_scenario_4.mp4

Proceeding after pedestrian crosses street

out_1_1.mp4

Block outside odor

out_1_7.mp4

Protect plant from direct sunlight

out_1_3.mp4

Trap candle's scent

out_1_5.mp4

Keep out unwanted gaze

out_1_4.mp4

Block out construction noise

ego_template_2_scenario_1.mp4

Slowing for lane closure congestion

ego_template_2_scenario_2.mp4

Yielding to overtaking ambulance

ego_template_2_scenario_3.mp4

Maintaining wary distance behind stopping truck

ego_template_2_scenario_4.mp4

Slowing to avoid SUV cut-in

ego_template_2_scenario_5.mp4

Complying with police speed enforcement

out_4_1.mp4

Clean dirty dishes

out_4_4.mp4

Thaw frozen vegetables

out_4_2.mp4

Water the plant

out_4_5.mp4

Washing mixed fruits in a bowl

out_4_8_2.mp4

Collect tap water

ego_template_3_scenario_5.mp4

Avoiding open door hazard

ego_template_3_scenario_2.mp4

Avoiding collision with elderly pedestrian

ego_template_3_scenario_3.mp4

Stopping abruptly due to highway accident

ego_template_3_scenario_4.mp4

Yielding safely to passing ambulance

ego_template_3_scenario_1.mp4

Braking for roadside assistance hazard

out_2_1.mp4

Improve air circulation and ventilation

out_2_3.mp4

Bid farewell and let guest out

out_2_4.mp4

Check delivered package

ego_template_4_scenario_2.mp4

Braking for occluded SUV at corner

ego_template_4_scenario_1.mp4

Stopping at a walking street

ego_template_4_scenario_3.mp4

Halting at unexpected police checkpoint

ego_template_4_scenario_4.mp4

Yielding to ambulance

ego_template_4_scenario_5.mp4

Picking up a passenger

out_6_4_open.mp4

1. Open the storage furniture

Store detergent out of children's reach

out_6_4_put.mp4

2. Put the detergent away

out_6_4_close.mp4

3. Close the storage furniture door

out_6_2_open.mp4

1. Open the storage furniture

Put away toys as guest arrive

out_6_2_put.mp4

2. Put away toys

out_6_2_close.mp4

3. Close the storage furniture door

ego_template_5_scenario_1.mp4

Yielding to ambulance

ego_template_5_scenario_2.mp4

Braking to avoid running a red light

ego_template_5_scenario_3.mp4

Stopping for a car running a red light

ego_template_5_scenario_4.mp4

Brake for a fallen box

ego_template_5_scenario_5.mp4

Halt for a high-speed police chase

out_8_2.mp4

Showcase an artwork

out_8_1.mp4

Brighten the desk to read a book

out_8_4.mp4

Setup workstation

ego_template_6_scenario_1.mp4

Abrupt lane change to avoid debris

ego_template_6_scenario_2.mp4

Overtaking slow vehicle

ego_template_6_scenario_3.mp4

Clearing path for ambulance with lane change

ego_template_6_scenario_4.mp4

Merging to open lane

ego_template_6_scenario_5.mp4

Swerving to avoid wrong-way driver

out_9_1.mp4

Remove glare on the screen

out_9_2.mp4

Sharing a presentation during a meeting

out_12_3.mp4

Grabbing a bucket to fill with sand

out_12_2.mp4

Fetching water for floor mopping

Google Sites

Report abuse