Scalable Continual Learning for Lifelong Foundation Models

For the pursuit of increasingly general intelligence, current foundation models are fundamentally limited by their training on static data, leading to outdated encoded information, saturation in knowledge accumulation, and wasteful use of compute resources. The increasing size of machine learning (ML) models puts ever more emphasis on scalable learning since even fine-tuning large models is becoming increasingly resource-intensive and time-consuming. Continual learning (CL) now emerges as a crucial framework in this new era, essential for dealing with the evolving scale and complexity of ML models. Yet, even the most recent methods in CL fall short of effectively addressing the challenges posed by the current data and compute scales. 

At this workshop, we discuss recent advances in scalable CL that could potentially replace static foundation model (FM) training, enabling us to model dynamic real-world information. We bring together experts and researchers from various domains, including language, vision, speech, and multimodal ML to exchange ideas and foster collaboration. With invited and contributed talks by distinguished researchers in the area, the workshop will delve into the evolving definition of CL, and how CL can enable the efficient development of foundation models. 

We welcome all contributions related to scaling the continual learning of foundation models. Potential areas of interest include but are not limited to:

• How should CL methods be utilized to avoid retraining large, foundation models?

• How can we address the challenge of catastrophic forgetting when fine-tuning FMs on considerably smaller and less diverse datasets in comparison to the extensive pretraining datasets?

• How can we address CL on a large scale when dealing with real-world problems with domain shifts and long-tailed data distributions?

• How can insights from other fields (online learning, meta-learning, reinforcement learning, neuroscience, AutoML, etc) inform and advance our CL of FMs?

• Does combining FMs with structured knowledge sources (databases, knowledge graphs, etc) help CL?

• What are the key considerations in designing benchmarks, evaluation protocols, and appropriate metrics for assessing CL of FMs?

• How can recent advances in FMs enhance CL techniques?

• What strategies can facilitate the seamless integration of CL and multi-modal learning systems?

Submission instructions 

To submit your paper, please consider the following instructions and guidelines.

• All contributions should be made via OpenReview. We welcome submissions of original, unpublished material, as well as work that is currently under review (i.e. has been submitted but not yet accepted elsewhere). Note that new OpenReview profiles created without an institutional email will go through a moderation process that can take up to two weeks while those created with an institutional email will be activated automatically.

• Page limit: Papers should be up to 4 pages, excluding references and supplementary materials.

• Template: Please use the NeurIPS 2024 style files. 

• Double blind reviews: authors should anonymize their submissions to ensure a double blind review process.

• LLMs policy: In the preparation of your contributions, the use of LLMs is allowed only as a general-purpose writing assist tool.

• You can submit (shortened versions of) main-track submissions to our workshop.

Publication. The workshop is non-archival. By default, accepted papers will be made publicly available on OpenReview. Authors can choose to opt-out if they do not wish for their work to be shared publicly.

Reviewing. Authors should nominate at least one person per contribution as a reviewer. The expected reviewing load is 3 papers.

Attending the workshop. Our workshop is primarily an in-person event, and authors are asked to present a poster at the workshop if possible. A subset of papers will be selected for presentation in short spotlight talks.

Tentative Timeline

Submission open: July 29, 2024, AoE.

Submission deadline: September 20, 23:59 AoE.

Decision notification: October 09, 2024, AoE.

For any questions, please contact continual-fomo@googlegroups.com.

Oral Presentations

• A Practitioner's Guide to Continual Multimodal Pretraining
  Karsten Roth, Vishaal Udandarao, Sebastian Dziadzio, Ameya Prabhu, Mehdi Cherti, Oriol Vinyals, Olivier J Henaff, Samuel Albanie, Matthias Bethge, Zeynep Akata

• Demystifying Language Model Forgetting with Low-Rank Example Associations
  Xisen Jin, Xiang Ren

• TiC-LM: A Multi-Year Benchmark for Continual Pretraining of Language Models
  Jeffrey Li, Mohammadreza Armandpour, Seyed Iman Mirzadeh, Sachin Mehta, Vaishaal Shankar, Raviteja Vemulapalli, Oncel Tuzel, Mehrdad Farajtabar, Hadi Pouransari, Fartash Faghri

• Exploring Continual Fine-Tuning for Enhancing Language Ability in Large Language Model
  Divyanshu Aggarwal, Sankarshan Damle, Navin Goyal, Satya Lokam, Sunayana Sitaram

• Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning
  Huiyi Wang, Haodong Lu, Lina Yao, Dong Gong