Time: Tuesday and Thursday, 3:30 - 4:50 PM
Location: Scaife Hall 105
Instructor David Held <dheld@andrew.cmu.edu>
TAs (to contact the TAs, please use Piazza - see below):
Hongyi Chen <hongyic@andrew.cmu.edu>
Yifan Sun <yifansu2@andrew.cmu.edu>
Ying Yuan <yingyua2@andrew.cmu.edu>
Jason Liu <jasonl6@andrew.cmu.edu>
Chaitanya Chawla <cchawla@cs.cmu.edu>
TA Office hours: TBD
Instructor Office hours: Available upon request - send an email to dheld@andrew.cmu.edu
Course Discussion / Announcements: https://piazza.com/cmu/spring2026/16831 (make sure to sign up to receive announcements)
We encourage you to ask questions on Piazza if you have any questions about a concept taught in class; you can even do so anonymously.
Robots need to make sequential decisions to operate in the world and generalize to diverse environments. How can they learn to do so? This is what we call the "robot learning" problem, and it spans topics in machine learning, deep learning, visual learning, and reinforcement learning. In this course, we will learn the fundamentals of topics in machine/deep/visual/reinforcement learning and how such approaches are applied to robot decision-making. We will study fundamentals of 1) machine/deep learning with an emphasis on approaches relevant to robotics; 2) reinforcement learning: model-based, model-free, on-policy (e.g., policy gradients), off-policy (e.g., Q-learning), offline, etc.; 3) imitation learning: behavior cloning, DAgger, inverse RL, etc.; and 4) visual learning geared towards decision making including topics like generative models and their use for robotics. The goal is for students to get both a high-level understanding of important problems in robot decision making as well as a low-level understanding of technical solutions.
By the end of the course, students should:
Be familiar with techniques of deep reinforcement learning and how they can be applied to robotics
Understand the principles behind different methods
Think about the tradeoffs between different approaches for solving robotics problems
Gain experience with using deep reinforcement learning algorithms
Grading will be as follows:
HW1: 15%
HW2: 15%
HW3: 15%
HW4: 15%
Project Presentation: 20%
Project Report: 20%
Participation: up to 2% extra credit - includes class or Piazza participation
For the homework assignments only (not the project), students will be allowed a total of 5 free late days to use throughout the semester.
You can submit any assignment up to a maximum of 3 days late. You can do this either by:
Using up to 3 of your free late days
If your late days are used up, you can submit it late (up to 3 days) for a 10% penalty per day.
If you submit the assignment more than 3 days late then you will get a 0 (regardless of late days).
If you have a medical situation, please make a private post on Piazza to ask for a medical extension. You do not need to explain the details of your medical situation.
You cannot use a fraction of a late day, (e.g., a one-hour late will be considered as a full late day).
Please note that the late days do not apply to any part of the final project. This policy will be enforced strictly.
Students in are absolutely encouraged to collaborate! Any assistance, though, must be limited to discussion of the concepts and sketching general approaches to a solution.
You are allowed to ask an LLM to help you clarify a concept from class!
In other words: GenAI tools such as ChatGPT are viewed as a collaborator in this course.
However, each student must write their own code and produce their own writeup.
Consulting another student's solution, solutions from the internet, or asking an LLM to write code for you, is prohibited on assignments.
These and any other form of collaboration constitute cheating.
If you work with someone on an assignment, please include their name in your write up and inside any code that has been discussed.
If we find highly identical write-ups or code without proper accreditation of collaborators, we will take action according to university policies.
You are not allowed to reuse code that you have written for other classes for this class. Please implement the required functions and code from scratch for this assignment.
You may not supply code or assignment writeups you complete during this class to other students in future instances of this course or make these items available (e.g., on the web) for use in future instances of this course (just as you may not use work completed by students who've taken the course previously).
If you have any question about whether some activity would constitute cheating, just be cautious and ask the instructor before proceeding!
If you have a disability and are registered with the Office of Disability Resources, I encourage you to use their online system to notify me of your accommodations and discuss your needs with me as early in the semester as possible. I will work with you to ensure that accommodations are provided as appropriate. If you suspect that you may have a disability and would benefit from accommodations but are not yet registered with the Office of Disability Resources, I encourage you to contact them at access@andrew.cmu.edu.
Reinforcement Learning: An Introduction by Richard Sutton
We can all benefit from support in times of stress. Studies have shown that your mental health can benefit by maintaining a healthy lifestyle by eating well, exercising, avoiding drugs and alcohol, getting enough sleep and taking some time to relax. These practices can help you achieve your goals and cope with stress.
Even with these practices, it is natural to struggle with mental health challenges. All of us benefit from support during times of struggle. You are not alone. There are many helpful resources available on campus and an important part of the college experience is learning how to ask for help. Asking for support sooner rather than later is often helpful. If you or anyone you know experiences any academic stress, difficult life events, or feelings like anxiety or depression, we strongly encourage you to seek support. Counseling and Psychological Services (CaPS) is here to help: call 412-268-2922 and visit their website at http://www.cmu.edu/counseling/. Consider reaching out to a friend, faculty or family member you trust for help getting connected to the support that can help.
If you need extra help for the course beyond what the instructor and TAs can provide, you can also use the Student Academic Success Center (SASC).
Academic Coaching--This program provides holistic, one-on-one peer support and group workshops to help undergraduate and graduate students implement habits for success. Academic Coaching assists students with time management, productive learning and study habits, organization, stress management, and other skills. Request an initial consultation here.
Peer Tutoring--Peer Tutoring is offered in two formats for students seeking support related to their coursework. Drop-In tutoring targets our highest demand courses through regularly scheduled open tutoring sessions during the fall and spring semesters. Tutoring by appointment consists of ongoing individualized and small group sessions.You can utilize tutoring to discuss course related content, clarify and ask questions, and work through practice problems. Visit the webpage to see courses currently being supported by Peer Tutoring.