The Thirteenth International Conference on Learning Representations (ICLR) 2025
Pengcheng Wang1*, Chenran Li1*, Catherine Weaver1, Kenta Kawamoto2, Masayoshi Tomizuka1, Chen Tang3†, Wei Zhan1
1 University of California Berkeley.
2 Sony Research Inc., Japan.
3Department of Computer Science, University of Texas at Austin.
* Equal Contribution, † Corresponding Author
Policies learned through Reinforcement Learning (RL) and Imitation Learning (IL) have demonstrated significant potential in achieving advanced performance in continuous control tasks. However, in real-world environments, it is often necessary to further customize a trained policy when there are additional requirements that were unforeseen during the original training phase. It is possible to fine-tune the policy to meet the new requirements, but this often requires collecting new data with the added requirements and access to the original training metric and policy parameters. In contrast, an online planning algorithm, if capable of meeting the additional requirements, can eliminate the necessity for extensive training phases and customize the policy without knowledge of the original training scheme or task. In this work, we propose a generic online planning algorithm for customizing continuous-control policies at the execution time which we call Residual-MPPI. It is able to customize a given prior policy on new performance metrics in few-shot and even zero-shot online settings. Also, Residual-MPPI only requires access to the action distribution produced by the prior policy, without additional knowledge regarding the original task. Through our experiments, we demonstrate that the proposed Residual-MPPI algorithm can accomplish the few-shot/zero-shot online policy customization task effectively, including customizing the champion-level racing agent, Gran Turismo Sophy (GT Sophy) 1.0, in the challenging car racing scenario, Gran Turismo Sport (GTS) environment.
Overview of the proposed algorithm: In each planning loop, we utilize the prior policy to generate samples and then evaluate them through both the log likelihood of the prior policy and an add-on reward to obtain the customized actions. Please find more details in our paper.
Gran Turismo Sport (GTS) is a realistic racing simulator with high-fidelity vehicle dynamics. GT Sophy 1.0, a DRL-based racing policy, has shown to outperform the world’s best drivers among the global players. Though GT Sophy 1.0 is a highly advanced system with superhuman driving ability, it tends to exhibit too aggressive route selection as the well-trained policy has the ability to keep stable on the simulated grass. However, in real-world racing, such behaviors would be considered dangerous and fragile because of the time-varying physical properties of real grass surfaces and tires. Therefore, we establish the task as to help the GT Sophy 1.0 drive on course while maintaining its superior racing performance.
We mainly present the demos of the following policies:
GT Sophy 1.0: A DRL-based racing policy who outperform the world’s best drivers among the global players. It serves as the prior policy for policy customization and the baseline to validate successful customization.
Residual-SAC: We consider Residual-SAC as another baseline to validate the advantage of online Residual-MPPI against RL-based solutions in the challenging racing problem.
Zero-shot Residual MPPI (Ours): We apply Residual MPPI with a given dynamics model pre-trained with the collected transition from the prior policy for Zero-shot policy customization.
Few-shot Residual MPPI (Ours): We also evaluate a Few-shot version of Residual MPPI , in which we iteratively update the dynamics with the customized planner's trajectories.
Zero-shot Residual-MPPI significantly enhances the safety of GT Sophy 1.0 and reduce its off-course steps. Further improvements are observed in Few-shot Residual MPPI who employing data gathered during the customization process to fine-tune the dynamics under a few-shot setting. In contrast, Residual-SAC yields a very conservative customized policy.
It is worth noting that over 80,000 laps of roll-outs are collected to achieve the current performance of Residual-SAC. In contrast, the data used to train GTS dynamics for Residual-MPPI, along with the online data for dynamics model fine-tuning, amounted to only approximately 2,000 and 100 laps respectively.
Case 1
Case 2
Case 3
Case 4
We visualize the difference between the actions selected by Residual-MPPI at each step compared to the original actions output by GT Sophy 1.0. As it can be seen, the additive actions are mostly zero when driving on a straight part of the trail.
