Introduction to Markov Decision Processes

By Martin L. Puterman and Timothy C. Y. Chan

We are currently writing a book that will be an accessible and up to date introduction to Markov decision processes (MDPs). Our target audiences include undergraduate students, graduate students and self-directed learners looking for a structured introduction that covers foundations, algorithms, and applications.

Drafts of certain chapters will be posted here as they become available. We welcome all feedback and suggestions. We believe Chapters 2-5 provide the necessary material for a course on MDPs.

This material will be published by Cambridge University Press as Introduction to Markov Decision Processes by Martin L. Puterman and Timothy C. Y. Chan. This pre-publication version of the following chapters is free to view and download for personal use only. Not for redistribution, resale, or use in derivative works. ©Martin L. Puterman and Timothy C. Y. Chan, 2023.