The course is open to BTech (4th year), Ph.D. and M.Tech.
Lectures:
Mon: 11 AM - 11.55 AM, Thurs: 9 AM-9.55 AM Fri: 10 AM -10.55 AM
Venue: 5G3 (Core 5)
Instructor:
Dr. Arghyadip Roy, Assistant Professor, MFSDS&AI, IIT Guwahati
Course Objective:
The course covers the fundamentals of Reinforcement Learning. The course starts with an introduction to Markov chains and Markov decision process as a prelude to Reinforcement Learning. This course is designed to provide theoretical as well as application-oriented perspectives. Case studies and examples covered will include many real-life applications from various domains. It is expected that the students will be able to apply Reinforcement Learning, the main take-away of this course, in their pursuits in machine learning and diverse application areas.
Course Content/ Syllabus:
Introduction to Reinforcement Learning (RL); Markov Chains: Discrete-time Markov chains, Stationary Distribution, Continuous-time Markov Chains. Markov Decision Process (MDP): Terminologies & Fundamentals; Finite Horizon MDP: Dynamic Programming (DP), Bellman Equation; Infinite Horizon Discounted Cost Problems: DP Equation, Fixed Point & Contraction Mapping, Value Iteration (VI), Policy Iteration (PI), Linear Programming formulation; RL Techniques: Multi-armed Bandit; Exploration & Exploitation, Optimism in the Face of Uncertainty, Upper Confidence Bound, Thompson Sampling; Monte-Carlo Methods: First-visit and Every-visit, Monte-Carlo Control, Off-policy Monte Carlo Control; Temporal Difference Learning: TD Prediction, Optimality of TD(0), SARSA, Q-learning, n-step TD Prediction, TD (lambda) algorithm, Linear Function Approximation, Linear TD (lambda); Policy Search Methods: Policy Gradient (PG) Theorem, REINFORCE, Variance Reduction in PG Methods, Actor-critic Methods.
Texts:
M.L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, 1st Edition, Wiley, 2005.
R.S. Sutton and A.G. Barto, Reinforcement Learning: An introduction, 2nd Edition, MIT Press, 2018.
References:
D.P. Bertsekas, Dynamic Programming and Optimal Control, Vol. I, 3rd Edition, Athena Scientific, 2005.
D.P. Bertsekas, Dynamic Programming and Optimal Control, Vol. II: Approximate Dynamic Programming, 4th Edition, Athena Scientific, 2012.
T. Lattimore and C. Szepesvári, Bandit Algorithms, 1st Edition, Cambridge University Press, 2020.
M. Harchol-Balter, Performance Modeling and Design of Computer Systems: Queueing Theory in Action, 1st Edition, Cambridge University Press, 2013.
Some more references: Lecture Notes by Lodewijk Kallenberg (Markov Decision Process), Shivaram Kalyanakrishnan (Foundations of Intelligent and Learning Agents), R. Srikant (Control of Stochastic Systems), Daniel J. Russo , Benjamin Van Roy , Abbas Kazerouni , Ian Osband and Zheng Wen (A Tutorial on Thompson Sampling) and Richard Weber (Optimization & Control).
Grading Scheme:
2 Quizes:20%
Midsem: 20%
Endsem: 40%
Course project (Modeling + Programing): 20%
Students auditing the course must score above 50% in the course to be awarded an audit grade.
Lecture Notes (Link):
Lecture 1-2: Introduction to Reinforcement Learning
Lecture 3: Probability Review & Discrete Time Markov Chain
Lecture 4: Stationary Distribution
Lecture 5: Ergodicity Theory
Lecture 6: Poisson Process & Exponential Distribution
Lecture 7: Continuous Time Markov Chain
Lecture 8: Markov Decision Process (MDP) Basics
Lecture 9: Finite Horizon Dynamic Programming
Lecture 10: Finite Horizon Stochastic Control
Lecture 11: Examples of Stochastic Dynamic Programming
Lecture 12: Infinite Horizon Discounted Cost MDP basics
Lecture 13: Existence of Optimal Policy
Lecture 14: Contraction mapping
Lecture 15: Value Iteration Algorithm
Lecture 16: Policy Iteration Algorithm
Lecture 17: Threshold Policy
Lecture 18: Multi-armed Bandit Basics
Lecture 19: Bandit Algorithms
Lecture 20: Upper Confidence Bound (UCB) Basics
Lecture 21: UCB Regret Analysis and KL-UCB Basics
Lecture 22: Thompson Sampling
Lecture 23: Monte Carlo Methods
Lecture 24: Off-policy Monte Carlo Methods
Lecture 25: TD Learning
Lecture 26: SARSA & Q Learning
Lecture 27: n-step TD Learning
Lecture 28: Function Approximation
Lecture 29: Policy Gradient Theorem
Lecture 30: REINFORCE
Lecture 31: Actor-critic & Batch RL