ECE524: Foundations of Reinforcement Learning

Spring 2024,     Tue. Thu. 09:30 am - 10:50 am, Equad B205, 

• Instructor:   Chi Jin                    Office hour:  Fri. 3:00-4:00 pm, Equad C332

• TA:                Wenhao Zhan          Office hour:  Mon. 3:00-4:00 pm, Friend Center 308. 

• Contents:     Mathematical foundations of RL, mostly about theorems and proofs.

• Grades:        5 problem sets (60%), 1 final exam (40%).

• No late homework. 

Lecture Notes

*Please see preliminary version of subscribed notes in 2020 version of the course.

• 1/30. Introduction [slides], MDP basics [note1][video1].

• 2/1. Bellman equations. [note2][video2]

• 2/6. Planning. [note3][video3]

• 2/8. Concentration inequalities. [note4][video4]

Problem Sets

• Problem set 1

Schedule (weekly basis)

Basics (tabular MDP): 

1. Intro, MDP basics and planning.

2. Concentration inequalities.

3. Generative models, value iteration.

4. Online RL, exploration, optimism.         [Homework 1 due]

5. Minimax lower bound.                                                 

6. Offline RL, pessimism.                                                       [Homework 2 due]

Advanced Topics:

1. Policy optimization.

2. Large state space, linear function approximation. [Homework 3 due]

3. General function approximation.                                                                     

4. Game theory and multiagent RL .            [Homework 4 due]

5. Learning Markov games.

6. Partial observable MDP.                                                         [Homework 5 due]

Reference Readings

• Reinforcement Learning: Theory and Algorithms (draft), by Alekh Agarwal, Nan Jiang, Sham M. Kakade, Wen Sun

• Reinforcement learning: an introduction, by Richard S. Sutton, Andrew G. Barto

• Algorithms for Reinforcement Learning, by Csaba Szepesvári

• Bandit Algorithms, by Tor Lattimore, Csaba Szepesvari

Mathematical Tools

• High dimensional probability. An introduction with applications in Data Science, by Roman Vershynin

• Concentration inequalities and martingale inequalities — a survey, by Fan Chung, Linyuan Lu 

Related Courses

• Nan Jiang, Statistical Reinforcement Learning

• Wen Sun and Sham Kakade, Foundations of Reinforcement Learning

• Dylan J. Foster and Alexander Rakhlin, Statistical Reinforcement Learning and Decision Making

• Alekh Agarwal and Alex Slivkins, Bandits and Reinforcement Learning

More practical/empirical version (will not be covered in this course)

• Sergey Levine, Deep Reinforcement Learning