A Comprehensive Benchmark for Evaluating External

and In-Cabin Risks

Under Review

code and dataset(Github)

Driving safety in assisted driving systems requires holistic understanding of both the external traffic environment and the internal driver state. However, existing research typically treats out-cabin scene understanding and in-cabin driver monitoring as separate problems, resulting in fragmented safety assessment and limited support for real-world warning applications. Moreover, conventional task-specific models often perform well on narrowly defined tasks but struggle to generalize to the diverse and long-tail hazards encountered in open-world driving scenarios. To address these limitations, we propose a new research setting, termed driving safety understanding for assisted driving, which aims to jointly perceive, reason about, and warn against risks by integrating information from both inside and outside the vehicle. To support this task, we construct a large-scale driving safety dataset containing 98K vision-language annotations that cover diverse in-cabin and out-cabin safety-critical scenarios, and further curate a 3K benchmark named DSBench (Driving Safety Benchmark) set for systematic evaluation. Based on this dataset, we develop DSVLM, a domain-specialized vision-language model for driving safety. Unlike end-to-end assisted driving models, DSVLM is designed as an in-vehicle safety assistant that identifies risk factors, infers potential danger, and provides timely warnings to help prevent accidents. Extensive evaluations against a wide range of mainstream open-source and proprietary VLMs show that existing models suffer substantial performance degradation in complex safety-critical situations, revealing a clear gap in reliable driving-safety understanding. In contrast, DSVLM establishes a new state of the art, achieving an average score of 68.4, which surpasses the strongest commercial baseline, Seed-1.6 (49.52), by 18.88 points. Its superiority is particularly pronounced on highly challenging tasks such as cockpit understanding, where DSVLM attains 80.1, dramatically outperforming the runner-up score of 29.49. These results demonstrate that DSVLM not only improves overall performance, but also delivers markedly stronger robustness and safety-awareness in complex real-world driving scenarios. The benchmark toolkit, code, and model checkpoints will be publicly accessible.

Overview

Data Collection Pipeline

The Overview of Our PPF-Tracker. First, we apply quasi-canonicalization to transform the pose tracking task into an SE(3)-invariant incremental prediction strategy for pose increment prediction problem. Then a voting scheme is used to predict coarse pose. Finally, the kinematic constrain is adopted to improve prediction reliability.

Details of DSBench

Performance Comparison of Various Models on Assisted Driving Tasks.

Appendix

Generalization Abilitys

To verify that our DSBench not only enhances existing models in the security domain but also boosts their general capabilities, we test the general performance of DSVLM on LingoQA. The experimental settings are consistent with those in the original paper. We compare the performance of DSVLM with several mainstream models. One additional advantage of selecting LingoQA to evaluate the generalization capabilities of our model lies in its original proposition: traditional metrics are inadequate for comprehensively assessing open-domain QA tasks. This aligns with our work's use of LLM-based scoring mechanisms in the metrics section. Table 4 further supports this observation, demonstrating that our model performs best under intelligent scoring scenarios but scores relatively low on traditional similarity-based metrics. This discrepancy underscores that our fine-tuned model exhibits a stronger reasoning capability. With the expansion of QA tasks in our training data, the diversity and richness of question-answer types have significantly increased, greatly enhancing the language expressiveness of DSVLM.

More Visualization

All prompt used in our work

Page updated

Google Sites

Report abuse