Learning to make good decisions is one of the keys to autonomous systems. This course will focus  on Reinforcement Learning (RL), a currently very active subfield of artificial intelligence, and it will discuss selectively a number of algorithmic topics including Markov Decision Process, Q-Learning, function approximation, exploration and exploitation, policy search, imitation learning, model-based RL and optimal control. This course provides both the foundations and techniques for developing RL and deep RL algorithms that interact with physical environments, and real application cases of RL will be introduced. Basic knowledge of machine learning and mathematical optimization are expected for this course.

Class: WF, E1-134, 1:30-2:50pm.

Instructor: Yize Chen, yizechen@ust.hk, Office: E3-604

TA: Yuxin Pan, yuxin.pan@connect.ust.hk

Schedule of classes

• Course Introduction [slides]

• Convex Function and Convex Optimization [slides][slides2]

• Optimization Problems and Algorithms [slides]

• Markov Decision Processes [slides]

• Model-Free Policy Evaluation [slides]

• PyTorch Tutorials [slides][Starter code]

Office hours

My office hours: Mondays 4:30pm-5:30pm, E3-604

TA office hour: Thursdays 6:30pm-7:30pm, E3-229

You are welcome to schedule other times by emailing me at yizechen@ust.hk.

Recommended Readings

• Optimization:

  • Boyd, Stephen, Stephen P. Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.

  • Sra, Suvrit, Sebastian Nowozin, and Stephen J. Wright, eds. Optimization for machine learning. Mit Press, 2012.

• Reinforcement Learning:

  • Sutton, Richard S., and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2018.

  • Grossi, Csaba. Algorithms for reinforcement learning. Springer Nature, 2022.

  • Bertsekas, Dimitri. Reinforcement learning and optimal control. Athena Scientific, 2019.

Homework Assignments

• Homework 1, due Feb 22nd, 11:59pm

Project

You are asked to carry out an original research project or derivation of theoretical works related to the course content and write a (roughly 6 page) report. 1-3 members form a group; one submission of report per group is expected. A poster presentation session will be organized by the end of the semester.

Submission instructions will come shortly.

Grades will be based on:

• Homework - 30%

• Midterm - 30%

• Final project presentation, report, and code - 40%

  • Poster Presentations May. 12

  • Project report due May. 19

• In-class bonus -5%