ECE524: Foundations of Reinforcement Learning

Fall 2025,     Tue. Thu. 9 - 10:20 am, Fine Hal 110, 

• Instructor:   Chi Jin                    Office hour:  Fri. 9 - 10 am, Equad C332

• TA:                Wenzhe Li         Office hour:   Wed. 4 - 5 pm, Equad D321

• Contents:     Mathematical foundations of RL, mostly about theorems and proofs.

• Grades:        5 problem sets (60%), 1 final exam (40%).

• No late homework. 

• See past versions of the course here: 2024 Verison, 2022 Version, 2020 Version

• Video recordings for lectures of the 2024 Version: YouTube.

Lecture Notes

• 9/2. Introduction [slides], MDP Model [note1].

• 9/4. Environment, Policy, Value [note2]

Problem Sets

Schedule (weekly basis)

Basics (tabular MDP): 

1. Intro, MDP basics and planning.

2. Concentration inequalities.

3. Generative models, value iteration.

4. Online RL, exploration, optimism.         [Homework 1 due]                                                 

5. Offline RL, pessimism.                                                       

6. Minimax lower bound. [Homework 2 due]

Advanced Topics:

1. Policy optimization.

2. Large state space, linear function approximation. [Homework 3 due]

3. General function approximation.                                                                     

4. Game theory and multiagent RL .            [Homework 4 due]

5. Learning Markov games.

6. Partial observable MDP.                                                         [Homework 5 due]

Reference Readings

• Reinforcement Learning: Theory and Algorithms (draft), by Alekh Agarwal, Nan Jiang, Sham M. Kakade, Wen Sun

• Reinforcement learning: an introduction, by Richard S. Sutton, Andrew G. Barto

• Algorithms for Reinforcement Learning, by Csaba Szepesvári

• Bandit Algorithms, by Tor Lattimore, Csaba Szepesvari

Mathematical Tools

• High dimensional probability. An introduction with applications in Data Science, by Roman Vershynin

• Concentration inequalities and martingale inequalities — a survey, by Fan Chung, Linyuan Lu 

Related Courses

• Nan Jiang, Statistical Reinforcement Learning

• Wen Sun and Sham Kakade, Foundations of Reinforcement Learning

• Dylan J. Foster and Alexander Rakhlin, Statistical Reinforcement Learning and Decision Making

• Alekh Agarwal and Alex Slivkins, Bandits and Reinforcement Learning

More practical/empirical version (will not be covered in this course)

• Sergey Levine, Deep Reinforcement Learning