Pre-requisites:

As such, there are no course-based pre-requisite requirements for this course. However, proficiency in the following would be needed to understand and follow the material of the course along with completing the assignment:

• Data Structures and Linear Algebra: You must understand basic data structure concepts. You should be comfortable taking (multivariable) derivatives and know matrix/vector notations and operations.

• Probability Basics: Familiarity with basic probability distributions (Continuous, Gaussian, Bernoulli, etc.) is expected. Further concepts such as Expectation, independence, probability distribution functions, and cumulative distribution functions must be clear to work-out the material in this course.

• Python coding: All assignments will involve coding in python. Hence, familiarity with the python programming language is required. Some assignments may also require the use of deep-learning libraries such as Keras.

For CS543, it is a prerequisite that either you have already credited CS533 or registered for CS533 in the current semester.

Course Credit Structure:

Both the courses (RL and RL Lab) are 500 level elective courses open to all the departments and all streams of B.Techs, M.Techs/M.Sc. and PhDs. The credit structure for the RL course is: 3-0-0-6-3 (3 Credits) and for RL Lab is: 0-0-2-1 (1 Credits). There will be 3 lectures (150 minutes) per week and 2 hrs of lab. Students are expected to invest 6+1 hrs of study time for this course per week. Don't get scared you do have 168 hrs/week to choose your study time from.

Reference Materials:

• Primary textbook - Reinforcement learning: An introduction. Sutton, Richard S., and Andrew G. Barto. 2nd Ed. MIT press, 2018. A draft version (not complete) of this book is available here

• Other reference books/materials

  • Markov Decision Processes by Martin L Puterman, Wiley.

  • Reinforcement Learning and Optimal Control by D. P. Bertsekas, Athena Scientific.

  • Research Papers

  • David Silver's video lectures from Google Deep Mind: https://www.youtube.com/playlist?list=PLqYmG7hTraZDM-OYHWgPebj2MfCFzFObQ (2015 Series)

  • Deep Mind 2021 Reinforcement Learning Lecture Series: https://www.deepmind.com/learning-resources/reinforcement-learning-lecture-series-2021

  • OpenAI Spinning up in Deep RL - https://spinningup.openai.com/en/latest/

Instructor:

Dr Shashi Shekhar Jha (shashi @iitrpr .ac.in)

Office: #215, 1st Floor, SRB

*We will use google classroom for all course related communications, personal emails to the instructor are discouraged unless the matter needs urgent and personal attention.

Teaching Assistants:

• Mr Shivam Kainth (shivam.20csz0006@iitrpr.ac.in)

• Mr Armaan Garg (armaan.19csz0002@iitrpr.ac.in)

Class Communication:

We will use Google Classroom for all course related communications including:

• Class announcements

• Reading material/ slides/ class notes

• Submission of assignments

• Submission of projects/demos

• Attendance

• Quizzes/Exams

To get the classroom code, email the course TAs.

Tentative Grading Policy:

• There will be four exams in the CS533 - RL course:

  • Quiz 1 - 15%

  • Mid-Sem Exam - 20%

  • Quiz 2 - 15%

  • End-Sem Exam - 20%

  • Class Engagement - 10% ( this weightage will be equally shared between CS533 and CS543)

• All exams will be pre-announced. The exact dates of quizzes will be available in the course schedule. Mid-sem and Exam-sem will be held as per the institute time-table. No requests will be entertained for the change in the schedule of quizzes. Further, there will not be any make-up quiz in the course.

• For the CS543 - RL Lab. course:

  • Programming Assignments (3) - 15%+15%+10%| There will be 3 programming intensive assignments. Each assignment may span over 2-3 weeks. Students should start working on the assignments as soon as they are announced.

• Course Project/Research Assignment (have weightage for both CS533 & CS543) :

  • Students can opt to work on a course project (post mid-sem) by presenting a convincing idea to the course Instructor and TAs. A call will be announced in the classroom for the student to submit their ideas. This can be done either individually or in a group of two. The evaluation of the projects will be scattered post the mid-sem exam. A final viva voce will be conducted at the end of the course for each project.

  • Students not opting (or not approved) for course project would need to work on a research based assignment i.e. Assignment 3. This assignment has to be done individually as other assignments.

  • Weightage : The course project/research assignment will be considered for 20% weightage in CS533 and for 50% weightage in CS534.

This is a tentative breakup of the grades and can change at the discretion of the Instructor. However, any change in grading policy will be duly intimated in advance.

• Getting a Pass Grade

In order to successfully clear the course, a student is expected to secure at least 40% of the total weightage in both the courses CS533 and CS534. If a student clears one course and fails in the other, s/he will be awarded an 'E' grade in both the courses.

• Course Audit

Those students who are auditing the course (only CS533 auditing is allowed) need to secure at least 40% of the total weightage (except the course project) in order to get a pass grade.

Academic Integrity and Honour Code:

• Any kind of plagiarism/cheating/copying etc. will attract an F grade in the course.

• Students are advised to read and understand about plagiarism and never indulge in the same.

• Students are encouraged to discuss and seek guidance (if needed) without breaking any academic integrity.

• All submissions ought to be the original work of the students.

• Any external source must be properly cited mentioning proper references.