Pretrained on extensive datasets through self-supervised learning, foundation models (FMs) have shown remarkable proficiency in transferring world knowledge to a wide array of downstream tasks. Owing to these capabilities, they hold significant potential to enhance decision-making agents. At the same time, these FMs are limited by their lack of grounding to specific tasks, skills, and environments. Nonetheless, FMs can help overcome the prevailing limitations of tabula rasa learning to improve sample efficiency, exploration, and generalization. 

This workshop investigates questions at the intersection of FMs and decision-making agents, encouraging critical debates on the potential and limitations of using FMs to enhance reasoning and learning. We will consider questions such as, but not limited to:

• How can FMs be leveraged in a sequential decision-making context, e.g., using large language models (LLMs) to improve performance of reinforcement learning algorithms?

•  What are the challenges of training FMs on data involving sequential decision-making?

• Can the knowledge contained in FMs help tabula rasa reinforcement learning in sparse feedback scenarios such as hard exploration tasks?

• How can the interactions between various modalities of multimodal FMs provide different views on a given decision-making problem?

Goal   The workshop aims to foster an inclusive environment where researchers and practitioners from all backgrounds can engage in discussions and build collaborations. More broadly, the workshop will focus on the following topics and problems:

• Enhancement: How can FMs be used for representation learning, sample efficiency, exploration, transfer learning, generalization, skill discovery, and other aspects of learning agents? Can the reasoning capabilities of FMs be enhanced by grounding them in sequential decision making data?

• Best practices: How to ensure scientific rigor and best practices working with FMs? What potential biases can occur from using FMs for decision making and how to assess them?

• Future directions and limitations: What are gaps in the current literature? Are current benchmarks appropriate?  What capabilities can we hope to see from FM-based agents in the future?

The workshop does not publish proceedings, and submissions are non-archival. Submission to this workshop does not preclude future publication. We encourage submissions of work in progress and completed work, and consider multiple formats and lengths. We will also consider submissions of tutorials through Google Colab that foster a more interactive environment.

We invite contributions on the following topic within the scope of training agents with FMs:

• Representation learning (e.g., spanning diverse modalities, embodiments, and environments);

• Sample efficiency in data-constrained environments (e.g. generating synthetic data);

• Improved exploration (e.g., narrowing the decision space with behavioral priors or prompting);

• Transfer learning and generalization (e.g., using general world knowledge);

• Skill discovery (e.g., by specifying self-supervised goals)

• World models and generative learning environments from internet-scale data;

• Benchmarks and evaluations for agent training;

• Applications papers that evidence the capabilities or limitations of traning agents with FMs;

• Other creative uses of FMs for training agents not considered here.

Submission Formats

• Short-paper track   This track considers short papers of 2-4 pages aimed at presenting early work or position papers. Upon acceptance, papers in this track will be required to submit a 4-page version providing additional technical details and reproducibility aspects.

• Full-paper track   This track considers of an 8-page paper for more mature work.  We invite submissions from papers available in public repositories without formal proceedings, such as arXiv. However, papers previously published in formal proceedings will not be considered.

• Tutorial track   Submissions for this track consist of tutorial notebooks that introduce or demonstrate the use of FMs to improve agent training. As FMs for decision-making are a new topic, notebooks should focus on the general RLC audience without expertise in FMs.  Tutorial submissions on OpenReview should contain an abstract with a link to a Google Colab or Kaggle notebook. They should include a clear and concise description of the users’ expected learning outcomes from the tutorial. Submissions of notebooks will also be anonymous. This track was inspired by the Tutorials Track of the Tackling Climate Change with Machine Learning Workshop, and we will use a similar template and instructions.  The template and detailed instructions are [here].

All accepted works will be published on our website and indexed by Google Scholar. All papers and tutorials will be submitted through OpenReview.  The discussion with the reviewers will not be made public. The reviewing process will be double-blind. 

Submission instructions

• Use the workshop's modified RLC LaTeX template on Overleaf: https://www.overleaf.com/read/qrbnjvbqvvqh#d2ead2

• Submit your contribution to the workshop's OpenReview portal: https://openreview.net/group?id=rl-conference.cc/RLC/2024/Workshop/TAFM

• Indicate the track as a keyword (short-paper track, full-paper track, or tutorial track)

• References and supplemental materials do not count toward the limit.

• Supplemental materials that contain enough details for reproducibility are encouraged. However, reviewers are not required to consider them during the revisions.

Virtual participation

While the workshop is primarily in-person, authors of accepted works who cannot attend in person will have the opportunity to present their work virtually.