CS533 & CS543
Reinforcement Learning Course & Lab.
About the Course:
Reinforcement Learning (RL) forms one of the core areas of artificial intelligence with relevance to a variety of domains that include robotics, game theory, control theory, multi-agent systems, swarm intelligence, and operations research. This course will provide a mathematical foundation for the field of reinforcement learning along with the core challenges and approaches, including generalization and exploration. Various latest advancements such as DeepRL, InverseRL, etc. will also be covered in this course.
The course will equip students with advanced skillset in the general area of artificial intelligence and prepare them to advance their careers (both research and professional) in the field artificial intelligence.
Pre-requisites:
As such, there are no course-based pre-requisite requirements for this course. However, proficiency in the following would be needed to understand and follow the material of the course along with completing the assignment:
Data Structures and Linear Algebra: You must understand basic data structure concepts. You should be comfortable taking (multivariable) derivatives and know matrix/vector notations and operations.
Probability Basics: Familiarity with basic probability distributions (Continuous, Gaussian, Bernoulli, etc.) is expected. Further concepts such as Expectation, independence, probability distribution functions, and cumulative distribution functions must be clear to work-out the material in this course.
Python coding: All assignments will involve coding in python. Hence, familiarity with the python programming language is required. Some assignments may also require the use of deep-learning libraries such as Keras.
For CS543, it is a prerequisite that either you have already credited CS533 or registered for CS533 in the current semester.
Course Credit Structure:
Both the courses (RL and RL Lab) are 500 level elective courses open to all the departments and all streams of B.Techs, M.Techs/M.Sc. and PhDs. The credit structure for the RL course is: 3-0-0-6-3 (3 Credits) and for RL Lab is: 0-0-2-1 (1 Credits). There will be 3 lectures (150 minutes) per week and 2 hrs of lab. Students are expected to invest 6+1 hrs of study time for this course per week. Don't get scared you do have 168 hrs/week to choose your study time from.
Reference Materials:
Primary textbook - Reinforcement learning: An introduction. Sutton, Richard S., and Andrew G. Barto. 2nd Ed. MIT press, 2018. A draft version (not complete) of this book is available here
Other reference books/materials
Markov Decision Processes by Martin L Puterman, Wiley.
Reinforcement Learning and Optimal Control by D. P. Bertsekas, Athena Scientific.
Research Papers
David Silver's video lectures from Google Deep Mind: https://www.youtube.com/playlist?list=PLqYmG7hTraZDM-OYHWgPebj2MfCFzFObQ (2015 Series)
Deep Mind 2021 Reinforcement Learning Lecture Series: https://www.deepmind.com/learning-resources/reinforcement-learning-lecture-series-2021
OpenAI Spinning up in Deep RL - https://spinningup.openai.com/en/latest/
Instructor:
Dr Shashi Shekhar Jha (shashi @iitrpr .ac.in)
Office: #215, 1st Floor, SRB
*We will use google classroom for all course related communications, personal emails to the instructor are discouraged unless the matter needs urgent and personal attention.
Teaching Assistants:
Mr Shivam Kainth (shivam.20csz0006@ iitrpr.ac.in)
Mr Rahul Narava (syam.21csz0018@ iitrpr.ac.in )
Lecture Schedule :
Classroom Room : CS1 (Slot PC2)
Mon - 10:00 - 11:00 AM
Tue - 10:00 - 11:00 AM
Wed - 10:00 - 11:00 AM
Lab Schedule:
Tuesday 2:00 PM [CS2]
Class Communication:
We will use Google Classroom for all course related communications including:
Class announcements
Reading material/ slides/ class notes
Submission of assignments
Submission of projects/demos
Attendance
Quizzes/Exams
To get the classroom code, email the course TAs.
Tentative Grading Policy:
There will be four exams in the CS533 - RL course:
Quiz 1 - 15%
Mid-Sem Exam - 20%
Quiz 2 - 15%
End-Sem Exam - 20%
Class Engagement - 10% ( this weightage will be equally shared between CS533 and CS543)
All exams will be pre-announced. The exact dates of quizzes will be available in the course schedule. Mid-sem and Exam-sem will be held as per the institute time-table. No requests will be entertained for the change in the schedule of quizzes. Further, there will not be any make-up quiz in the course.
For the CS543 - RL Lab. course:
Programming Assignments (3) - 15%+15%+10%| There will be 3 programming intensive assignments. Each assignment may span over 2-3 weeks. Students should start working on the assignments as soon as they are announced.
Course Project/Research Assignment (have weightage for both CS533 & CS543) :
Students can opt to work on a course project (post mid-sem) by presenting a convincing idea to the course Instructor and TAs. A call will be announced in the classroom for the student to submit their ideas. This can be done either individually or in a group of two. The evaluation of the projects will be scattered post the mid-sem exam. A final viva voce will be conducted at the end of the course for each project.
Students not opting (or not approved) for course project would need to work on a research based assignment i.e. Assignment 3. This assignment has to be done individually as other assignments.
Weightage : The course project/research assignment will be considered for 20% weightage in CS533 and for 50% weightage in CS534.
This is a tentative breakup of the grades and can change at the discretion of the Instructor. However, any change in grading policy will be duly intimated in advance.
Getting a Pass Grade
In order to successfully clear the course, a student is expected to secure at least 40% of the total weightage in both the courses CS533 and CS534. If a student clears one course and fails in the other, s/he will be awarded an 'E' grade in both the courses.
Course Audit
Those students who are auditing the course (only CS533 auditing is allowed) need to secure at least 40% of the total weightage (except the course project) in order to get a pass grade.
Academic Integrity and Honour Code:
Any kind of plagiarism/cheating/copying etc. will attract an F grade in the course.
Students are advised to read and understand about plagiarism and never indulge in the same.
Students are encouraged to discuss and seek guidance (if needed) without breaking any academic integrity.
All submissions ought to be the original work of the students.
Any external source must be properly cited mentioning proper references.
Counselling Support:
Any student experiencing any mental or emotional stress can seek the free and confidential clinical counselling by contacting the Counselling Cell of IIT Ropar.
Deepak Kr. Phogat (Clinical Psychologist, Counselor)
Common Office: 01, Medical Centre, Utility Block, Main Campus
Email: deepak.phogat @ iitrpr ac in
Phone: 01881-24-2264
Bhawna Suri (Counseling Psychologist, Counselor)
Email: bhawna @ iitrpr ac in
Phone: 01881-24-2261
Additionally, students can also approach the Snehlita Wellbeing cell under Student Affairs section for support and counselling