Foundation Models for Decision Making @NeurIPS 2023

Foundation models pretrained on diverse vision and language datasets have demonstrated exceptional capabilities in performing a wide range of downstream vision and language tasks. As foundation models are deployed in real-world applications such as dialogue, autonomous driving, healthcare, and robotics, they inevitably face new challenges such as learning from external feedback, adapting to different task modalities, and performing long-term reasoning and planning. Such challenges have traditionally been at the core of sequential decision making, encompassing areas such as reinforcement learning, imitation learning, planning, search, and optimal control. These research fields have traditionally focused on task-specific settings with limited prior knowledge, and yet there has been significant research progress in surpassing human performance in tasks like playing board games and Atari video games, as well as operating robots to complete navigation and manipulation tasks. However, since these methods generally learn to solve a specific task from scratch without broad knowledge from vision and language, they can struggle with generalization and sample efficiency. 

Research in the intersection of foundation models and sequential decision making is gaining attention. Research in foundation models has expanded to address long-term reasoning and multiple model interactions, while researchers in sequential decision making are developing larger datasets and training larger-scale interactive agents. Further blurring the lines between the two fields, dialogue agents have been optimized by reinforcement learning with human feedback, and large pretrained vision-language models have been used as perception and reasoning components of embodied agents. Foundation models have also been adapted to interact with search engines, calculators, translators, simulators, and program interpreters. Despite these early successes, foundation models for decision making still faces many scientific questions and challenges that have not been addressed by existing work. Examples of questions that we hope to make progress towards answering through this workshop include: 

1. Develop language model agents that can automatically learn to interact with humans, tools, the world, and each other in a scientific and principled way.

2. Derive sound, practical, and scalable algorithms similar to RLHF and MCTS for language and vision based decision making applications.

3. How to structure environments and tasks so that vision language foundation models can benefit traditional decision making applications in control, planning, and reinforcement learning?

4. Foundation models are trained on data without actions. How to overcome this limitation from both the dataset and modeling perspectives?

Goal of the workshop. The goal of this workshop is to bring together the sequential decision making community including planning, search, RL, and optimal control, together with the foundation models community in vision and language to confront the challenges in decision making at scale. The workshop will span high-level discussions on how foundation models and decision making can benefit each other when jointly considered and low-level algorithmic details of various decision making algorithms and vision-language architectures, which might lead to both opportunities or challenges. More specific topics will include but are not limited to:

1. Foundation model agents interacting with humans, computers, tools, simulators, physical world, and each other.

2. Rethinking the implementation, ecosystem, and model modularity of decision making agents under emerging technologies such as ChatGPT and language model plug-ins.

3. Applying foundation models to traditional decision making problems in control, planning, online / offline RL.

4. Learning multi-modal, multi-task, multi-environment, and generalist policies.

5. Long-horizon reasoning and planning in language models.

6. New evaluation protocols, benchmarks, datasets, and applications that apply foundation models to solve decision making problems.

7. Theoretical understanding of the roles foundation models play in decision making.