Previous Names: Course was formely known as "Statistical Techniques in Robotics" though it has been updated to teach the latest methods in deep reinforcement learning
Time: Tuesday and Thursday, 1:25 - 2:45 PM
Location: NSH 1305
Instructor David Held (to contact the instructor, please use Piazza - see below)
TA: Shuli Jiang and Sam Triest (to contact the TA, please use Piazza - see below)
TA Office hours: Tuesdays from 6 - 7 PM in NSH 4201 (Shuli); Thursdays from 12:15 - 1:15 PM in NSH 4513 (Sam)
Instructor Office hours: Available upon request
Course Discussion / Announcements: https://piazza.com/cmu/fall2022/16831 (make sure to sign up to receive announcements)
We encourage you to ask questions on Piazza if you have any questions about a concept taught in class; you can even do so anonymously.
Robots need to make sequential decisions to operate in the world. How can they learn to do so? In this course, we will learn the fundamentals of deep reinforcement learning and how such approaches can be applied to robot decision making. We will learn about:
Reinforcement learning: Including multi-armed bandits (UCB, Thompson Sampling, Gaussian Processes), policy gradients (TRPO, PPO, GAE, PAL/MAL), off-policy reinforcement learning (importance sampling, DDPG, SAC, MBPO), model-based RL (CEM, MPPI; iLQR);
Imitation learning: Including BC, DAgger, and IRL (LP-IRL, MaxEntIRL, GAIL) and offline RL (AWAC / CRR, IQL, CQL, MOPO);
Sim2Real Transfer: How to transfer policies from simulation to the real world (GAN, VAE, SimOpt, BayesSim)
Throughout the course, we will look at many examples of how such methods can be applied to real robotics tasks.
Familiarity with Linear Algebra, Calculus, Probability, and Python
Grading will be as follows:
Assignments: 80%
Project: 20%
There will be 5 assignments in total; each will be worth 16% of the final grade
Letter grade cut-offs will depend on relative performances of students and are only approximate:
Near or above 100%, at the instructor's discretion - A+
Above 92% - A
90-92% - A-
80-90% - B-, B, or B+
70-80% - C-, C, or C+
60-70% - D-, D, D+
Less than 60% - Fail
Information about the assignments are described in the "assignments" tab at the top of this page.
Information about the project is described in the "project" tab at the top of this page.
Reinforcement Learning: An Introduction by Richard Sutton
Become familiar with techniques of deep reinforcement learning and how they can be applied to robotics
Understand the principles behind different methods
Think about the tradeoffs between different approaches for solving robotics problems
Gain experience with using deep reinforcement learning algorithms
For this course, I will be recording class sessions and making them available to you for your personal, educational use. Recordings of class sessions are covered under the Family Educational Rights and Privacy Act (FERPA) and must not be shared with anyone outside your course-section. The purpose of these recording is so students in this course (and only students in this course) can re-watch past class sessions. Feel free to use the recordings if you would like to review something we discussed in class or if you are temporarily unable to attend class.
You are encouraged to work on assignments together BUT you must do your own work. If you work with someone on an assignment, please include their name in your write up and inside any code that has been discussed. If we find highly identical write-ups or code without proper accreditation of collaborators, we will take action according to university policies.
We can all benefit from support in times of stress.Studies have shown that your mental health can benefit by maintaining a healthy lifestyle by eating well, exercising, avoiding drugs and alcohol, getting enough sleep and taking some time to relax. These practices can help you achieve your goals and cope with stress.
Even with these practices, it is natural to struggle with mental health challenges. All of us benefit from support during times of struggle. You are not alone. There are many helpful resources available on campus and an important part of the college experience is learning how to ask for help. Asking for support sooner rather than later is often helpful. If you or anyone you know experiences any academic stress, difficult life events, or feelings like anxiety or depression, we strongly encourage you to seek support. Counseling and Psychological Services (CaPS) is here to help: call 412-268-2922 and visit their website at http://www.cmu.edu/counseling/. Consider reaching out to a friend, faculty or family member you trust for help getting connected to the support that can help.
All people have the right to be addressed and referred to in accordance with their personal identity. Students are encouraged to use NameCoach to indicate the name that they prefer to be called and to identify pronouns with which they would like to be addressed. I will do my best to address and refer to all students accordingly and encourage everyone in the class to do so as well.
We must treat every individual with respect. We are diverse in many ways, and this diversity is fundamental to building and maintaining an equitable and inclusive campus community. Diversity can refer to multiple ways that we identify ourselves, including but not limited to race, color, national origin, language, sex, disability, age, sexual orientation, gender identity, religion, creed, ancestry, belief, veteran status, pregnancy, or genetic information. Each of these diverse identities shape the perspectives our students, faculty, and staff bring to our campus. We at CMU will work to promote diversity, equity and inclusion. We acknowledge our imperfections while we also fully commit to the work, inside and outside of our classrooms, of building and sustaining a campus community that increasingly embraces these core values.
Each of us is responsible for creating a safer, more inclusive environment. Unfortunately incidents of bias or discrimination do occur, whether intentional or unintentional. They contribute to creating an unwelcoming environment for individuals and groups at the university. Therefore, the university encourages anyone who experiences or observes unfair or hostile treatment on the basis of identity to speak out for justice and support, within the moment of the incident or after the incident has passed. Anyone can share these experiences using the following resources:
Center for Student Diversity and Inclusion: csdi@andrew.cmu.edu, (412) 268-2150
Report-It online anonymous reporting platform: www.reportit.net username: tartans password: plaid
All reports will be documented and deliberated to determine if there should be any following actions. Regardless of incident type, the university will use all shared experiences to transform our campus climate to be more equitable and just.
The course instructors also encourage anyone who experiences or observes unfair or hostile treatment to report such incidences to the course instructors so that we can address such situations.
If you are worried about affording food or feeling insecure about food, there are resources on campus who can help. Email the CMU Food Pantry Coordinator to schedule an appointment:
Pantry Coordinator
412-268-8704 (SLICE office)
If you have a disability and are registered with the Office of Disability Resources, I encourage you to use their online system to notify me of your accommodations and discuss your needs with me as early in the semester as possible. I will work with you to ensure that accommodations are provided as appropriate. If you suspect that you may have a disability and would benefit from accommodations but are not yet registered with the Office of Disability Resources, I encourage you to contact them at access@andrew.cmu.edu.
It is my intent that students from all diverse backgrounds and perspectives be well served by this course, that students’ learning needs be addressed both in and out of class. It is my intent to present materials and activities that are respectful of diversity: gender, sexuality, disability, age, socioeconomic status, ethnicity, race, and culture. Your suggestions are encouraged and appreciated. Please let me know ways to improve the effectiveness of the course for you personally or for other students or student groups. In addition, if any of our class meetings conflict with your religious events, please let me know so that we can make arrangements for you.
If you need extra help for the course beyond what the instructor and TAs can provide, you can also use the Student Academic Success Center (SASC).
Academic Coaching--This program provides holistic, one-on-one peer support and group workshops to help undergraduate and graduate students implement habits for success. Academic Coaching assists students with time management, productive learning and study habits, organization, stress management, and other skills. Request an initial consultation here.
Peer Tutoring--Peer Tutoring is offered in two formats for students seeking support related to their coursework. Drop-In tutoring targets our highest demand courses through regularly scheduled open tutoring sessions during the fall and spring semesters. Tutoring by appointment consists of ongoing individualized and small group sessions.You can utilize tutoring to discuss course related content, clarify and ask questions, and work through practice problems. Visit the webpage to see courses currently being supported by Peer Tutoring.