Versatile Scene-Consistent Traffic Scenario Generation as Optimization with Diffusion

Versatile Behavior Diffusion

The model encompasses a multi-modal behavior predictor and a multi-agent diffusion policy for realistic and controllable simulation. The scenario context tokens are encoded through a query-centric Transformer-based encoder. The predictor generates prior distributions of potential behaviors for individual agents in the scene. These marginal behavior priors or other structured forms can be used to guide the multi-agent diffusion policy to generate desired and interactive scenarios.

Generate multi-agent scene-consistent and interactive scenarios with VBD

All (up to 32) agents in the scene are controlled by the VBD model.

f9bac9d901209ef4.mp4

Interaction at stop sign

ad301ce3a54ae2bd.mp4

Cruise

9629721da46ca2b3.mp4

Complex intersection

2392d88912cffcd0.mp4

Interaction with side vehicles

cec70b4f834af3f6.mp4

Busy intersection

8c18267600a702b2.mp4

Interaction at narrow road

Closed-loop reactive simulation with VBD

SDC (gray) is controlled by a log-playback planner and other agents (blue) are controlled by the VBD model.

7a17b9e1b2ba281e.mp4

React to SDC lane changing

c2ee8b7c450b0055.mp4

Interact with SDC at narrow road

bc2d48816fbc902.mp4

Interact with SDC at stop sign

5e46df17c3738631.mp4

Overtake SDC

5d200d35188ff424.mp4

Interact with SDC at left turn

12858b7f79387840.mp4

Interact with SDC at stop sign

Scenario editing with the VBD model's predicted behavior priors

A human user selects the target agent and its target goal from the VBD model's predicted behavior priors. The model's diffusion policy generates scene-consistent scenarios with that condition.

video.mp4

Nominal sample

video.mp4

Prior-guided diffusion

Edit: Agent 5 speeds up to pass first

video.mp4

Nominal Sample

video.mp4

Prior-guided diffusion

Edit: Agent 1 goes straight

Generate safety-critical scenarios using VBD

We use two guidance strategies for generating safety-critical scenarios in diffusion policy. In game guidance, the green agent is the Avoider, the purple agent is the Adversarial Pursuer, and the blue agents are the others. In prior guidance, the green agent is the Ego and the purple agent is the Adversarial.

video.mp4

Nominal Sample

video.mp4

Game-guided safety-critical sample

ec5fd750cfde767c.mp4

Prior-guided safety-critical sample

video.mp4

Nominal Sample

video.mp4

Game-guided safety-critical sample

9c7064be6796ae04.mp4

Prior-guided safety-critical sample

video.mp4

Nominal Sample

video.mp4

Game-guided safety-critical sample 1

video.mp4

Game-guided safety-critical sample 2

video.mp4

Nominal Sample

video.mp4

Game-guided safety-critical sample

8a3f8dd35c4180d4.mp4

No conflict goals found in behavior priors