Embedded Files
BEVLM: Distilling Semantic Knowledge from LLMs into Bird's-Eye View Representations
Summary
Summary
The integration of Large Language Models (LLMs) into autonomous driving has attracted growing interest for their strong reasoning and semantic understanding abilities, which are essential for handling complex decision-making and long-tail scenarios. However, existing methods typically feed LLMs with tokens from multi-view and multi-frame images independently, leading to redundant computation and limited spatial consistency that hinders accurate 3D spatial reasoning. On the other hand, Bird's-Eye View (BEV) representations learned from geometrically annotated tasks (e.g., object detection) provide spatial structure but lack the semantic richness of foundation vision encoders. To bridge this gap, we propose BEVLM, a framework that connects a spatially consistent and semantically distilled BEV representation with LLMs. Through extensive experiments, we show that BEVLM enables LLMs to reason more effectively in cross-view driving scenes, improving accuracy by 46% by leveraging BEV features as unified inputs. Furthermore, by distilling semantic knowledge from LLMs into BEV representations, BEVLM significantly improves closed-loop end-to-end driving performance in safety-critical scenarios across UniAD and VAD, with gains of up to 28.2%.
Motivation
Motivation
Vision Language Models (VLMs) leverage large-scale image-text pre-training for semantic richness, but extract visual tokens independently from multi-view and multi-frame images, failing to model spatial consistency for 3D spatial reasoning. Bird's-Eye View (BEV) representations provide a compact and spatially consistent top-down view of the 3D environment, but cannot be pre-trained at scale on semantically rich image-text data. BEVLM bridges this gap by connecting a spatially consistent and semantically distilled BEV representation with LLMs, uniting the semantic richness of VLMs with the spatial consistency of BEV.
Spatial Representation Study
Spatial Representation Study
RQ1: Can BEV representations be aligned with the language space?
RQ1: Can BEV representations be aligned with the language space?
We pass BEV features from the UniAD encoder through a learned projector into an LLM, and compare against a task-specific decoder whose outputs are converted to answers via rule-based logic on binary object-existence questions from DriveLM-nuScenes. BEVLM substantially outperforms the majority-class and linear probe baselines and approaches UniAD's detection head, with InternVL3-1B reaching 90.8% versus UniAD's 92.8%, while scaling the LLM to 8B yields 95.3% and surpasses the detector itself. This shows that a simple MLP projector can effectively map BEV features into the language space while preserving meaningful spatial and semantic cues.
RQ2: Can BEV representations enable LLMs to perform better spatial reasoning?
RQ2: Can BEV representations enable LLMs to perform better spatial reasoning?
We compare three visual representations: I_ViT (tokens from the original VLM's ViT), I_UniAD (image-backbone tokens before BEV fusion), and B_UniAD (BEV tokens after fusion). On single-view DriveLM object-existence questions, B_UniAD already outperforms both image-based inputs, and on the cross-view Ego3D object-centric questions the advantage becomes pronounced, improving MCQ accuracy by 46.0% and decreasing L1 error by 27.8%. Notably, B_UniAD matches I_ViT even though the ViT encoder is 10x larger, confirming that BEV significantly improves spatial reasoning, especially for panoramic scene understanding.
Semantic Distillation
Semantic Distillation
RQ3: Can we combine the spatial consistency of BEV and the semantic richness of LLM?
RQ3: Can we combine the spatial consistency of BEV and the semantic richness of LLM?
We frame the LLM as a fixed semantic teacher that supervises the BEV encoder (student) through high-level VQA tasks, forcing the BEV grid to encode safety-relevant scene information, while jointly training with object detection to preserve its spatial structure. This representation-level distillation yields a semantically-enhanced BEV encoder that, when plugged into the UniAD and VAD end-to-end pipelines, delivers consistent closed-loop gains in safety-critical scenarios, with up to 28.2% higher NeuroNCAP safety score and a lower collision rate.
Team
Team
Thomas Mornninger*, Mercedes-Benz Research & Development North America Inc
Shaoyuan Xie*, Ph.D. student, University of California, Irvine
Alfred Chen, Assistant Professor, University of California, Irvine
Sihao Ding, Mercedes-Benz Research & Development North America Inc
* means equal contribution; the order is determined alphabetically.
Acknowledgement
Acknowledgement
This work was primarily supported by Mercedes-Benz Research & Development North America, Inc.
Shaoyuan Xie and Qi Alfred Chen were supported in part by (1) the National Science Foundation under grants CNS-2145493 and CNS-2413877, and (2) the U.S. Department of Transportation under Grant 69A3552348327 through the CARMEN+ University Transportation Center.
Google Sites
Report abuse