Upcoming session

Register here

YouTube Livestream: here

Tutorial Abstract: Meta-learning in deep neural networks (Julian)

This tutorial introduces the meta-learning framework in deep neural networks, which enables rapid adaptation to novel tasks using limited data. We will explore the advantages of meta-learning and its applications in various domains. Furthermore, we will discuss the relationship between meta-learning and the emergent phenomenon of in-context learning in large language models (LLMs). We will draw parallels between these two approaches and investigate the ability of LLMs to perform meta-learning entirely within the context of the input data, a concept known as meta-in-context learning.

Research Talk: Learning (to learn) useful representations in a structured world (Jane)

Meta-learning, the ability to learn the right structure or prior knowledge to facilitate new learning, is heavily reliant on structured data. Humans, deep RL agents, and even large language models (LLMs) are all capable of meta-learning. While recurrent neural network-based models can be linked to neural activations in biological organisms, understanding how LLMs perform this quick, in-context learning is more difficult. LLMs can be pre-trained on human-generated artifacts, such as the internet and books, which contain substantial structure and enable good generalization. New approaches have been introduced that allow us to more closely interrogate how they work, approaches directly taken from the cognitive sciences. In this talk I discuss how we can better understand both deep RL agents and LLMs by looking at structure within their training data through this lens, and why they’re so powerful.

About the Seminar

With the advent of foundation models and the emergence of capabilities at scale, algorithms designed to improve the training and inference efficiency of machine learning systems have become critical research topics. Despite their importance, expertise in training the most efficient systems is often in short supply, limiting the ability of teams to scale experiments, work at maximum productivity, develop foundation models for underexplored domains, and make the best use of our computing resources. 

This seminar series seeks to educate, provide a platform for, and facilitate collaborative research opportunities for all algorithmic and technological questions arising in the quest for more efficient machine learning at scale. The desired outcomes of the seminar series are:

• A proliferation of knowledge on state-of-the-art approaches directly employable by diverse research teams hoping to scale training or make use of large-scale models.

• The provision of a research discussion forum on emerging topics to help catalyze further research and form connections among researchers invested in researching these questions. 

Format

Each session is organized in a unique co-hosted format, consisting of a 30-minute tutorial presented by a rising star researcher followed by a 50-minute talk on recent work and open questions by an established scientist in the field. Beyond providing an accessible entry point to the topic of discussion, the Rising Star Series is intended as an inclusive element, with presenters drawn from the speaker's lab or broader research community. We conclude with a 10-minute Q&A with the main speaker. 

Upcoming Speakers

• Ari Morcos (DatologyAI) & Andreas Kirsch (University of Oxford)

• Joaquin Vanschoren (TU Eindhoven)

• Razvan Pascanu (Google DeepMind)

• Song Han (MIT)

• Torsten Hoefler & Langwen Huang (ETH Zürich)

• Vijay Janapa Reddi (Harvard University)

Past sessions

• March 25th: Nicolas Papernot (U of Toronto) & Anvith Thudi (U of Toronto) on 

"Machine Unlearning" Recording available here.

• March 11th: Sara Hooker (Cohere) & Valentina Pyatkin (A2I & University of Washington) on 

"Large Language Models" Recording available here.

Access

In the spirit of an open, diverse, inclusive, and collaborative scientific culture, we will livestream the seminar series on YouTube, allowing anyone (regardless of affiliation) to follow the seminar online. In addition, we will allow remote participation through live questions on Zoom and the use of online Q&A tools. All participants are required to abide by the ICML Code of Conduct.

Organization

Seminar Hosts: Jonathan Richard Schwarz, Owen Queen, Marinka Zitnik

Supported by: 

Department of Biomedical Informatics at Harvard Medical School, Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University, and Harvard Data Science Initiative.