Large Language Models as Decision Makers for Autonomous Driving

Abstract

Existing learning-based autonomous driving (AD) systems face challenges in comprehending high-level information, generalizing to rare events, and providing interpretability. To address these problems, this work employs Large Language Models (LLMs) as a decision-making component for complex AD scenarios that require human commonsense understanding. We devise cognitive pathways to enable comprehensive reasoning with LLMs, and develop algorithms for translating LLM decisions into actionable driving commands. Through this approach, LLM decisions are seamlessly integrated with low-level controllers by guided parameter matrix adaptation. Extensive experiments demonstrate that our proposed method not only consistently surpasses baseline approaches in single-vehicle tasks, but also helps handle complex driving behaviors even multi-vehicle coordination, thanks to the commonsense reasoning capabilities of LLMs. This paper presents an initial step toward leveraging LLMs as effective decision-makers for intricate AD scenarios in terms of safety, efficiency, generalizability, and interoperability. We aspire for it to serve as inspiration for future research in this field.

Demo

We showcased our system's performance in various complex scenarios. The pink vehicle represents the ego vehicle, the grey circle signifies the sensing range, green vehicles are those sensed, blue ones are unsensed, and red vehicles are the ones LLM is concerned about.

Reasoning ability

This example showcases LLM's ability to understand and reason with high-level information, affirming the effectiveness of our chain-of-thought approach. The combination of attention allocation, situational awareness, and action guidance ensures that our system consistently exhibits the correct driving behavior.

Multi-vehicle Joint Control

Each vehicle is individually controlled by a distributed LLM, with one central LLM acting as the "brain" of the fleet for multi-vehicle communication and coordination. Each distributed LLM reports the situation it is into the central LLM and receives commands to control the ego vehicle; the central LLM judges and gives the coordination commands based on the environmental information and the reports from the distributed LLMs.

Textual guidance

Our approach enables users or utilizes high-precision maps to provide textual instructions that guide the AD system's decision-making process. We conducted an experiment involving a road construction scenario. Upon receiving textual guidelines, our approach successfully recognized the situation and gave appropriate driving behavior.

Driving style adjustment

Our approach simplifies the process of driving style adjustment by merely providing textual descriptions to the LLM through a dedicated interface. When there is low risk of overtaking, LLM instructed to drive aggressively will make reasonable overtaking decisions, while those directed to drive conservatively will opt to slow down and follow the vehicle in front of it.

Page updated

Google Sites

Report abuse