Paper presentations and projects: schedule & sign up sheet
Deadlines:
Oct 6 - project proposals due
TBA - final project submission due
Class 1 (Wed, Sept 3, 3:30-5:30pm)
Lecturer: Irina Rish
Topic: Intro and Overview: A brief history of AI at Scale (slides, video)
Papers: The Bitter Lesson, GPT-3 paper: Language Models are Few-Shot Learners
Class 2 (Mon, Sept 8, 3:30-5:30pm)
Lecturer: Irina Rish
Topic: Intro and Overview: Continual Learning at Scale (slides, video)
Class 3 (Wed, Sept 10, 3:30-5:30pm)
Lecturer: Irina Rish
Topic: Overview of Papers to Present and Some Projects Topics (video-part1, video-part2 )
Class materials: some of the previous Topics & Papers (focus on: Continual Learning at Scale, Alignment and Safety, Emergence, Phase Transitions and Stat Physics of ML), Some previous large-scale projects, Towards Time_Series Foundation Models
Class 4 (Mon, Sept 15, 3:30-5:30pm)
Part 1: Lecturers: Alireza Dehghanpour Farashah and Aditi Khandelwal
Topic: Scaling Laws for Neural Language Models (slides, video)
Also covered: Training Compute-Optimal Large Language Models (Chinchilla Explained: video), Emergent Abilities of Large Language Models, Are emergent abilities of LLMs a Mirage? Additional materials: Neural Scaling Laws and GPT-3 (video); a nice overview of the history of scaling laws: Scaling Laws for LLMs: from GPT-3 to o3
Part 2: Lecturer: Hiroki Naganuma
Topic: An Empirical Model of Large-Batch Training (slides, video)