Embedded Files
Project Objective
Project Objective
We aim to develop a platform to validate the effectiveness of passenger-centric CAV controls by integrating pre-trained LLMs with a VR environment.
Project Incentives
Project Incentives
Utilizing VR to conduct CAV experiments provides three unique advantages:
Improved Safety
Improved Safety
VR simulation can eliminate risks for all personnel involved when experimenting with CAVs, while providing an immersive environment close to a real-world driving experience. For example, in the current project, VR allows us to gauge a passenger's riding experience in a CAV without compromising their safety.
Reduced Cost
Reduced Cost
The cost of VR headsets on the market ranges from $299 to $3,499 [1], which is less than the average cost of a new vehicle in 2024 at $47,000 [2]. Furthermore, the simulated roads can be customized to the exact needs of the experiment, and there is no limit on the maximum number of test vehicles, which will dramatically reduce the experiment cost.
Expedited Research Time
Expedited Research Time
VR can facilitate rapid turnaround between two experiments, leading to a shorter research and development cycle. This has great advantages in both the academia and the industry [3].
Project Architecture
Project Architecture
The IDS VR-LLM testbed is developed in conjunction with the existing work of CARLA Simulator [4], including various map layouts, traffic light system, and CAV routing system. By making use of CARLA's Python API, this proposed approach allows more versatility when adding new features (such as VR), better integration with existing machine learning models (LLMs), and easier adaptation by the research community.
- User
As the centerpiece of the project, the user will input their preferences to the system in the format of natural language.
- LLM
The LLM will interpret the user input and convert it into parameters for the objective function of the control problem.
- Control Algorithm
The optimal control problem will be solved using a numerical algorithm (e.g. optimal control, model predictive control, etc.) to generate feasible trajectories for the CAV, saving its results into a CSV file for global access.
- CSV
This CSV file bridges between the control algorithm and the CARLA Python API, serving as a live database to update CAV trajectories based on user preferences.
- Python API
The Python API will read the CAV trajectory values from the CSV file, then utilizes CARLA's traffic manager to reflect the changes and adjust the driving style of the CAV in real-time.
- Carla Simulation
The CARLA simulation hosts the entire experiment map, where it connects to multiple Python scripts simultaneously to adjust the simulation parameters or spawn vehicles and pedestrians.
- VR Headset
Using CARLA's Python API and pygame library [5], CARLA's simulation rendering can be projected into a 360-degree VR environment, allowing users to experience in the CAV riding as realistically as possible.
Project Demonstration
Project Demonstration
Stereoscopic VR Rendering
Stereoscopic VR Rendering
Carla External Vehicle Behavior Control
Carla External Vehicle Behavior Control
This video clip showcases the utilization of CARLA's traffic management system through external control inputs and a global CSV file. While this demonstration does not utilize LLM, it shows how the user can input simple commands to alter the behaviors of the CAV (e.g. go through a red light under emergency situation), thus customizing the riding experience.
References:
[1] T. Alsop, “VR headset comparison by price 2024,” Statista, 03-Jun-2024. [Online]. Available: https://www.statista.com/statistics/1337123/vr-headset-comparison-by-price/. [Accessed: 01-Aug-2024]
[2] T. Krisher and T. A. Press, “The average price of a new car is over $47,000-but analysts see prices dropping all throughout the rest of the year,” Fortune, 28-Feb-2024. [Online]. Available: https://fortune.com/2024/02/28/how-expensive-new-used-cars-outlook-forecast/. [Accessed: 02-Aug-2024]
[3] P. G. Smith and D. G. Reinertsen, Shortening the Product Development Cycle, May-1992. [Online]. Available: https://strategy2market.com/wp-content/uploads/2014/05/Shortening-Product-Development-Cycle.pdf. [Accessed: 03-Aug-2024]
[4] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “Carla: An open urban driving simulator,” arXiv.org, 10-Nov-2017. [Online]. Available: https://arxiv.org/abs/1711.03938. [Accessed: 02-Aug-2024]
[5] Real Python, “PyGame: A Primer on game programming in python,” Real Python, 18-Jul-2023. [Online]. Available: https://realpython.com/pygame-a-primer/. [Accessed: 03-Aug-2024]
[6] V. Aksoy, I. Ratmansky, and A. Watson, “How does Oculus link work? the architecture, pipeline and AADT explained,” Developer Center, 22-Nov-2019. [Online]. Available: https://developer.oculus.com/blog/how-does-oculus-link-work-the-architecture-pipeline-and-aadt-explained/. [Accessed: 03-Aug-2024]
[7] W. Antonelli and T. Lyles, “Which VR headsets work with steam? here’s a full guide to virtual reality on the steam gaming app,” A Full Guide to Which VR Headsets Work with Steam, 19-Jul-2024. [Online]. Available: https://www.businessinsider.com/guides/tech/which-vr-headsets-work-with-steam. [Accessed: 03-Aug-2024]
Image Courtesy :
Title Image: Class VR, upscaled by imgupscaler.com.
Project Icon: Open AI and CARLA Simulator.
Project Architecture Flow Chart: Flaticon, Texmesonet, Wikimedia, CARLA.
Author of the Page: Simon Tian
Edited By: Aditya Dave
Page updated
Report abuse