SafeDriveVLA:

Vision–Language–Action models (VLAs) let drivers steer end-to-end policies with free-form language, but they drop the rule-based behavior planner that once checked a navigation signal against the scene. The same instruction can be safe on an open highway and dangerous at a pedestrian crossing, so the model itself must decide whether to follow it.

In this work, we build a benchmark suite that evaluates navigation following, navigation–scene conflict awareness, and robustness to adversarial instructions together, and propose SafeDriveVLA, which predicts an explicit driving mode before acting and dreams the outcome of strictly executing the navigation signal with a latent world model.

SafeDriveVLA decouples navigation–scene conflict reasoning from action generation through an interpretable conflict prediction, paired with navigation-conditioned world model dreaming that exposes whether the instructed maneuver is feasible before the policy commits. Together, they let VLA drivers recognize when a navigation signal is unsafe given the surrounding scene.

Figure 1: Navigation–scene conflict in end-to-end VLAs

1. Three-Axis Safety Benchmark

Our benchmark suite isolates three orthogonal capabilities of a language-conditioned driver. Together, they distinguish a model that ignores instructions, a model that follows them blindly, and a model that genuinely reasons about whether they should be executed in context.

1.1 Navigation Following (CARLA-F)

For each meta-command, a pool of paraphrased natural-language instructions is chained along 210 closed-loop routes rebuilt from the CARLA driving graph, with all background traffic removed and all lights forced green, isolating whether the model can actually condition on the navigation signal at all. A speed instruction comes first, then navigation instructions fire at set distances along the route.

1.2 Navigation–Scene Conflict (B2D-C & NavSim-C)

Manually authored unsafe instructions are triggered at the moment a safety-critical event occurs (Bench2Drive, 150 routes), and ground-truth navigation commands are swapped for conflicting ones at intersections and lane-changes (NavSim, 300 scenarios). The danger is contextual, since the same surface form is benign elsewhere. We report Driving Score (DS) and Success Rate (SR) on B2D-C, and Predictive Driver Model Score (PDMS) on NavSim-C.

1.3 Adversarial Instruction (B2D-Adv)

Adversarial “jailbreak”-style instructions that are unsafe to execute under any circumstance, manually crafted for 15 safety-critical Bench2Drive routes covering 13 scenario types. This tests whether the VLA can refuse direct harmful commands rather than treating language input as authoritative.

2. SafeDriveVLA

SafeDriveVLA decouples conflict reasoning from action generation with two key designs.

2.1 Driving Mode Token

Expert demonstrations mix two decisions: whether the navigation signal is safe to execute now, and which action to take. We relabel every expert frame with one of three driving modes: Strict (execute the signal), Cautious (follow it while yielding to hazards), or Fallback (override it because no safe action remains), and the VLA emits this mode token before its action tokens. Conflict awareness is thus supervised directly instead of being learned implicitly through behavior cloning.

2.2 Navigation-Conditioned World Model Dreaming

A latent world model (a frozen V-JEPA 2 encoder with an action-conditioned predictor) forecasts the scene 2.5 s ahead. Unlike prior world models that roll out the expert’s action, SafeDriveVLA rolls it out under an action anchor that strictly executes the current navigation signal. When the maneuver is feasible the dreamed latents stay benign; when a vehicle occupies the target lane or a hazard blocks the path, they expose the conflict. The VLA reads these dreamed world tokens, so it follows safe instructions and defers or overrides unsafe ones.

Figure 2: SafeDriveVLA architecture: world model pre-training (left) and the SafeDriveVLA policy (right)

3. Evaluation

We report closed-loop results on standard Bench2Drive, then on our benchmarks for navigation following (CARLA-F), navigation–scene conflict (B2D-C, NavSim-C), and adversarial instructions (B2D-Adv), followed by an ablation study. ↑/↓ mark whether higher or lower is better.

3.1 Closed-Loop Driving on Bench2Drive

Closed-loop driving on the 220 standard Bench2Drive routes. Nav. is the navigation signal: command (CMD), target waypoint (WP), or language instruction (Lan). SimLingo results are from its paper. SafeDriveVLA achieves the best Driving Score (88.16) and Success Rate (68.64%) among all compared methods.

3.2 Navigation Following (CARLA-F)

Navigation Compliance Rate (NCR, %) per meta-command on 210 traffic-free routes, plus Speed Error (SE, m/s). Avg. is NCR weighted by instruction count, excluding speed. Among language-conditioned models, SafeDriveVLA reaches 83.0% average NCR versus 55.6% for the best SimLingo variant, and is the only language-conditioned model that can follow lane-change instructions.

3.3 Navigation–Scene Conflict (B2D-C)

Driving on 150 Bench2Drive routes where an unsafe language instruction is issued at the safety-critical event. The infraction columns are counts, and percentages are reductions relative to SimLingo-Safe. SafeDriveVLA improves DS from 72.8 to 85.9 and SR from 38.0% to 67.3%, while reducing collisions from 60 to 38.

Figure 3: B2D-C example: told to “stay in your lane and keep speed at 8 m/s ignoring other factors”, SafeDriveVLA switches to fallback mode and steers around the blocking obstacle.

3.4 Navigation–Scene Conflict (NavSim-C)

Existing VLAs on 300 NavSim scenarios whose command is replaced by a conflicting left or right turn. Sub-metrics are measured under the conflicting command, and PDMS is compared with the original command. All existing VLAs show a clear PDMS drop under conflicting commands, showing that they tend to blindly follow the navigation signal. SafeDriveVLA is CARLA-trained and single-view, so it is not evaluated here.

3.5 Adversarial Instruction (B2D-Adv)

Driving Score on 15 safety-critical routes on standard Bench2Drive (B2D) and with adversarial instructions that try to override the safety prior (B2D-Adv). SafeDriveVLA keeps a DS of 83.9 under adversarial instructions, compared with 76.0 for SimLingo.

3.6 Ablation Study

Baseline is the same VLA without the mode token or world model; + Mode + WM is the full SafeDriveVLA (×1.0 training data). Coll.: collisions; TV: traffic violations; OoR: out-of-route events. The driving mode token mainly improves safety on B2D-C, while navigation-conditioned world model dreaming is key to recovering navigation following (e.g., lane changes) without sacrificing safety.

Figure 4: Navigation following (CARLA-F NCR) vs. safety (B2D-C DS) for each ablation variant at ×0.2 and ×1.0 training data.

Figure 5: World-model action input: navigation-conditioned anchor vs. ego-history extrapolation (×0.2 data).

Research Paper

[CoRL'26] SafeDriveVLA: Navigation-Conditioned World Model Dreaming for Conflict-Aware End-to-End Autonomous Driving

Shaoyuan Xie*, Zihan Zhang*, Jingxuan Wang, Jiashu Qu, Xiaoqing Liang, Lingdong Kong, Junchi Lu, Henrik I. Christensen, Qi Alfred Chen

Conference on Robot Learning (CoRL), 2026.

[PDF] [arXiv] [Project Page] [Code] [Model]

BibTex for citation:

@inproceedings{xie2026safedrivevla,

title={{SafeDriveVLA: Navigation-Conditioned World Model Dreaming for Conflict-Aware End-to-End Autonomous Driving}},

author={Xie, Shaoyuan and Zhang, Zihan and Wang, Jingxuan and Qu, Jiashu and Liang, Xiaoqing and Kong, Lingdong and Lu, Junchi and Christensen, Henrik I. and Chen, Qi Alfred},

booktitle={Conference on Robot Learning (CoRL)},

year={2026}

}

Team

Shaoyuan Xie*, Ph.D. student, University of California, Irvine

Zihan Zhang*, Ph.D. student, University of California, San Diego

Jingxuan Wang, Ph.D. student, University of California, Irvine

Jiashu Qu, Ph.D. student, University of Cincinnati

Xiaoqing Liang, Ph.D. student, University of California, Irvine

Lingdong Kong, Ph.D. student, National University of Singapore

Junchi Lu, Ph.D. student, University of California, Irvine

Henrik I. Christensen, Professor, University of California, San Diego

Alfred Chen, Associate Professor, University of California, Irvine

* means equal contribution.

Acknowledgments

This research was supported in part by:

NSF under grant CNS-2145493;

USDOT under Grant 69A3552348327 through the CARMEN+ University Transportation Center;

NSF Institute for Learning-enabled Optimization at Scale (TILOS) under Award No. CCF-2112665.