Large Language Models (LLMs) took us by storm with the launch of ChatGPT and GPT-4, following the development of BERT, T5, and GPT. ChatGPT and many LLMs adopted RL from human feedback (RLHF) for human alignment, and they excel at several NLP tasks out of the box such as machine translation, summarization, and much more. At the same time, RL has had remarkable achievements like playing Go, Starcaft II, and Gran Turismo. Beyond games, RL helped with magnetic control of tokamak plasmas, navigating stratospheric balloons, matrix multiplication and sorting. We believe there is a huge untapped potential for the marriage of these two fields. In particular, LLMs can further benefit from the RL framework for:

• Planning: RL can empower LLMs to evaluate and optimize sequences of actions over time, enabling more coherent and effective planning. By receiving feedback on their planning decisions, LLMs can iteratively refine their strategies to excel in task-oriented scenarios. Consequently, this continuous feedback loop ensures that LLMs become progressively adept at handling complex, multi-step tasks and user interactions, leading to more goal-directed interactions.

• Exploration: The RL framework can facilitate adaptive exploration in LLMs, allowing them to weigh the benefits of generating an immediate response versus seeking additional user input. By constantly adjusting to feedback, LLMs can optimize this trade-off, ensuring they query the user when ambiguity exists and confidently generate answers when sufficient information is available. This iterative learning process ensures that LLMs become increasingly proficient at navigating user interactions, striking a balance between proactive responses and seeking clarifications. 

• Personalization:  Using RL, LLMs can dynamically adapt to individual user preferences and behaviors, honing their outputs to align with specific user needs over time. By analyzing feedback and rewards tied to the user satisfaction, LLMs can iteratively refine their responses, ensuring they resonate more closely with individual users. This continuous adaptation allows LLMs to deliver highly personalized interactions, fostering a more tailored and engaging user experience.

While RL community started to bring LLM advances on their side (e.g. Decision Transformer, Trajectory Transformer), LLMs provide unparalleled opportunities to enhance RL further: 

• Rich Representation: LLMs can help RL by encoding intricate environmental nuances into comprehensive state representations, addressing the challenges of non-Markovian situations where current states may not hold all the relevant information. Furthermore, leveraging their vast knowledge, LLMs can equip RL agents with sparse data situations boosting zero/few-shot learning.

• Explainability: LLMs can serve as interpreters for RL agents, transforming complex policy decisions into intuitive, human-readable narratives. By distilling the underlying logic and rationale of an agent's actions, LLMs make it easier for users to trust and collaborate with AI systems. Furthermore, LLMs facilitate a two-way dialogue, allowing humans to query and understand the motivations, strategies, and potential future actions of RL agents in dynamic environments.

• Task Decomposition: LLMs can aid RL agents in task decomposition by breaking down high-level goals into more manageable sub-tasks using their deep semantic understanding. LLMs can provide structured insights and hierarchies, allowing RL agents to approach problems in a modular fashion and optimizing solutions for individual components. 

The goal of our workshop is to bring together RL and LLM communities to facilitate cross-pollination. We will discuss possible opportunities including but not limited to the above areas. Furthermore we will have practitioners share their success stories of working on RL&LLMs problems, and the insights gained from such applications.

⏰️ Schedule 

Accepted Papers

1. Suspicion-Agent: Playing Imperfect Information Games with Theory of Mind Aware GPT-4 [pdf 📄]
   Guo, Jiaxian*; Yang, Bo; Yoo, Paul; Lin, Yuchen; Iwasawa, Yusuke; Matsuo, Yutaka

2. Generate Subgoal Images before Act: Unlocking the Chain-of-Thought Reasoning in Diffusion Model for Robot Manipulation with Multimodal Prompts [pdf 📄]
   Ni, Fei*; Hao, Jianye; Wu, Shiguang; Longxin, Kou; Liu, jiashun; Zheng, Yan; Wang, Bin; Zhuang, Yuzheng

3. CriticGPT: Multimodal LLM as a Critic for Robot Manipulation [pdf 📄]
   Liu, Jinyi*; Yuan, Yifu; Hao, Jianye; Ni, Fei; Fu, Lingzhi; Chen, Yibin; Zheng, Yan

4. Decision Transformer With Tokenized Actions [pdf 📄]
   Annett, Graham*; Andersen, Timothy

5. Reinforcement Learning for Optimizing RAG for Domain Chatbots [pdf 📄]
   Kulkarni, Mandar*; Tangarajan, Praveen; Kim, Kyung; Trivedi, Anusua

6. Software Security Vulnerability Repair Using Reinforcement Learning with Large Language Models [pdf 📄]
   Islam, Nafis Tanveer*; Karkevandi, Mohammad Bahrami; Najafirad, Peyman

7. Exploring Reinforcement Learning with Large Language Models for Enhancing Badminton Players' Strategies [pdf 📄]
   Wang, Kuang-Da; Wang, Yung-Chien*; Chien, Yen-Che; Peng, Wen-Chih Chris; Bo Zhou, Hsieh; Yong En, Tian

8. DeLF: Designing Learning Environments with Foundation ModelsGuo, Jiaxian*; Yang, Bo; Yoo, Paul; Lin, Yuchen; Iwasawa, Yusuke; Matsuo, Yutaka [pdf 📄]
   Afshar, Aida*; Li, Wenchao