Fall 2023

WORLD KNOWLEDGE IN THE TIME OF LARGE MODELS

This talk will discuss the massive shift that has come about in the vision and ML community as a result of the large pre-trained language and language and vision models such as Flamingo, GPT-4, and other models. We begin by looking at the work on knowledge-based systems in CV and robotics before the large model revolution and discuss the impact it had. This impact can be broken down into three areas in which world knowledge should be studied in the context of these new models: evaluation, harnessing large models, and building outside knowledge. First, evaluating world knowledge is even more important as the large model revolution gives more easy access to world knowledge. Next, we discuss recent work in harnessing models such as Flamingo and Chinchilla for visual and procedural knowledge. Finally, the talk discusses how, by focusing on knowledge acquisition as an agent-centric problem, we can make developments in retrieving and collecting world knowledge.

Reaching for the stars with indigenous

What I will speak to are the Calls to Action from the Truth and Reconciliation Commission of Canada. In particular, these calls tackle improvements in environmental, economic, and educational issues and opportunities as well as support for language and cultural preservation and enrichment. Basically, a respect for and inclusion of Indigegogy across society. I will share stories from my own personal and professional interests and reflect on how my unique combination of expertise and experience can benefit the ways of doing things going forward. What I bring as one of only a few female First Nation Indigenous chemists, but also as an educator, community leader, and mother - is a means to empower learners and educators. My hope is that science and STEM can become more appealing to Indigenous children, youth, girls, and women, and overall a more inclusive, more diverse, and more progressive field for the benefit of everyone, and the world’s future.

Bringing active learning to an industrial scale

Active learning is a useful machine learning technique that has emerged as a promising approach to tackle the challenge of limited labeled data in machine learning. It allows for more efficient use of resources by selecting the most informative data points for labeling, resulting in cost savings in data annotation.

The remarkable performance of deep neural networks often depends on the availability of massive labeled training data. To alleviate the load of data annotation with labels, deep active learning aims to sample a minimal set of training points to be labelled which yields maximal model accuracy. 

In the first part of the talk, I will cover an efficient active sampling criterion to sample data for annotation, which automatically shifts from an exploration type of sampling to a class-decision-boundary refinement. Our criterion relies on a process of diffusing the existing label information over a graph constructed from the hidden representation of the data. We analyze our sampling criterion and its exploration - refinement transition in light of the eigen-spectrum of the diffusion operator. Additionally, we provide a comprehensive sample complexity analysis that captures the two phases of exploration and refinement.

In the second part of the talk, I will show how diffusion in latent space of Variational Auto-Encoder can also be used as a filter to reduce the computational effort of processing large unlabeled sets in every step of active label acquisition. This 10x acceleration allows us to scale active learning to industrial scale while allowing continuous data annotation by humans.  If tie allows, I will show a demo of an industrial scale efficient active-learning-based annotation system we have built at Bell Labs that relies on our diffusion algorithms.

Limitations of large language models

Large language models (LLMs) are becoming increasingly used in various downstream applications not only in natural language processing but also in various other domains including computer vision, reinforcement learning, and scientific discovery to name a few. This talk will focus on the limitations of using LLMs as task solvers. What are the effects of using LLMs as task solvers? What kind of knowledge can an LLM encode (and also what it cannot encode)? Can they efficiently use all the encoded knowledge while learning a downstream task? Are LLMs susceptible to the usual catastrophic forgetting while learning many tasks? How do we identify the biases that these LLMs encode and how do we eliminate those biases? In this talk, I will present an overview of several research projects in my lab that attempt to answer all these questions. This talk will bring to light some of the current limitations of LLMs and how to move forward to build more intelligence systems.