The reading group
The aim of the reading club is to provide a bi-weekly space to learn about RL and its intersection with LLMs (more on this below). Our current set up is sessions where a speaker can explain a topic in this area, followed up by a friendly discussion. The idea is to shape the format of the reading club based on people's preferences and interests!
Topics
The goal of this reading group is to explore the intersection of reinforcement learning and large language models, from foundations to cutting-edge research. We plan on beginning with some core RL concepts (MDPs, policy gradients, value functions) and LLM fundamentals (transformers, pretraining, scaling laws), then progress to contemporary topics including RLHF, RL approaches to reasoning and planning, and so on.
Suitable for researchers and students with basic ML background interested in understanding how RL techniques are shaping the next generation of language models. We'll discuss key papers, share insights from recent work, and explore open problems in using RL to enhance model reasoning capabilities.
You can suggest topics here.
Where?
Every other Thursday at 1pm at Huxley 410.
How can I join?
Email me or Sergio Estan Ruiz.
Session 1 (Tom Coates): LLMs 101
Abstract: "What is this transformer thing, anyway?" I will describe what is going on inside a Large Language Model, starting from fundamental definitions.
Session 2 (Tom Coates): LLMs 102
Abstract: Finishing the talk from session 1.
Session 3 (Sara Veneziale): Training and deploying LLMs
In this talk, we will go through a high-level overview of how modern LLMs are trained. We’ll highlight the different stages of training, focusing on the different ways that engineers can influence model behaviour at both training and inference stages, and pointing at where resource constraints (either on the data or the compute side) become an issue.
Session 4 (Paul Lezeau): Training a small language model to speak fake Latin
Lab session! The goal of this session is for everyone to get their hands dirty and actually have a go at implementing what we've covered so far with a toy problem.
Notebooks:
Beginner version (we'll go through this together): click here (Google Colab link).
Trickier version (if you're already familiar with PyTorch and want to try this out on your own): click here (Google Colab link).
Solution: click here (Google Colab link). Note: the solution presented here is definitely not optimal - feel free to improve on it!
Session 5 (Benjamin Sanati): Intro to RL
In this talk, I will show how probability theory, statistical inference, and decision theory are used to describe agents that learn to act by interacting with their environment. After a philosophical overview, we will consider decision-making and learning under uncertainty, building toward the fundamental principles of reinforcement learning.
Session 6 (Sergio Estan): Safety research in AI: are we screwed?
I just received a book called "If anyone builds it, everyone dies. The case against AI superintelligence". That seems a bit intense. After all, LLMs are just regression models trained on huge amounts of garbage from the internet, right? So, to what extent do new LLM-based systems pose a real threat? And perhaps more importantly, what can we do to minimise such risks? In this talk I'll try to explain the safety research landscape surrounding LLMs, talking about ways in which LLM-system can be attacked and some defence mechanisms that practitioners and researchers put in place. Some of the talk will be based on the following AI safety online course which I really recommend: https://bluedot.org/courses/technical-ai-safety/1/1 .