The model encompasses a multi-modal behavior predictor and a multi-agent diffusion policy for realistic and controllable simulation. The scenario context tokens are encoded through a query-centric Transformer-based encoder. The predictor generates prior distributions of potential behaviors for individual agents in the scene. These marginal behavior priors or other structured forms can be used to guide the multi-agent diffusion policy to generate desired and interactive scenarios.
Our VBD paper won the best paper award at the Workshop on Autonomous Vehicles Across Scales (AVAS) @ RSS 2024. Here is the recording of our spotlight talk.
Our VBD model won the second-place award at the 2024 WOD Sim Agents Challenge. Here is our technical report.
All (up to 32) agents in the scene are controlled by the VBD model.
Interaction at stop sign
Cruise
Complex intersection
Interaction with side vehicles
Busy intersection
Interaction at narrow road
SDC (gray) is controlled by a log-playback planner and other agents (blue) are controlled by the VBD model.
React to SDC lane changing
Interact with SDC at narrow road
Interact with SDC at stop sign
Overtake SDC
Interact with SDC at left turn
Interact with SDC at stop sign
A human user selects the target agent and its target goal from the VBD model's predicted behavior priors. The model's diffusion policy generates scene-consistent scenarios with that condition.
Nominal sample
Prior-guided diffusion
Edit: Agent 5 speeds up to pass first
Nominal Sample
Prior-guided diffusion
Edit: Agent 1 goes straight
We use two guidance strategies for generating safety-critical scenarios in diffusion policy. In game guidance, the green agent is the Avoider, the purple agent is the Adversarial Pursuer, and the blue agents are the others. In prior guidance, the green agent is the Ego and the purple agent is the Adversarial.
Nominal Sample
Game-guided safety-critical sample
Prior-guided safety-critical sample
Nominal Sample
Game-guided safety-critical sample
Prior-guided safety-critical sample
Nominal Sample
Game-guided safety-critical sample 1
Game-guided safety-critical sample 2
Nominal Sample
Game-guided safety-critical sample
No conflict goals found in behavior priors