16-881: Deep Reinforcement Learning for Robotics

Spring 2020

Course Info:

Days TR

Room NSH 3002

Time 4:30 - 5:50 PM

Instructor David Held (Office hours upon request)

TA Victoria Dean (Office hours: Mondays 3:30-4:30pm in Smith 214; also may be available upon request)

Instructor+TA Email List 16881-tas@lists.andrew.cmu.edu

Course Discussion / Announcements: https://piazza.com/cmu/spring2020/16881 (make sure to sign up to receive announcements and to submit assignments)


Note that class on April 9 is cancelled - there will be a makeup class on Friday April 3 from 10 to 11:20 AM in Gates 4303.

Course Description

Each class, 2 papers will be presented; one paper will be a paper on deep RL; the other will be a paper on a similar topic / achieving a similar task, which may or may not use deep RL. The class will read and discuss these papers and try to understand:

  • What are the strengths and limitations of the approach described in each paper?

  • What insights can we take away from each paper?

  • How can the method described in each paper be improved?

Students will also work on a class project related to RL of their choosing.

Class timing

The timing for the class will be as follows:

  • 23 minutes for presentation 1 (+4 minutes for discussion and changeover)

  • 23 minutes for presentation 2 (+4 minutes for discussion)

  • 10 minute small group discussion

  • 16 minute class discussion

Prerequisites

Students are expected to have already have a basic understanding of reinforcement learning, such as from 10-703, 16-748, 16-831, or a similar course, prior to taking this course. If you are not sure if your background is sufficient, please post on Piazza to ask. If you need a refesher, I highly recommend Reinforcement Learning: An Introduction by Richard Sutton and Andrew Barto, especially Chapters 2-6 and 13. Also, the second lecture of the course will cover a quick review of reinforcement learning.

Grading

  • Presentations: 30%

  • Paper reviews: 40%

  • Class project: 30%

Educational Outcomes

  1. Become familiar with some classic robotics papers and their approaches

  2. Become familiar with some recent deep reinforcement learning papers

  3. Think about the tradeoffs between different approaches for solving a robotics problem

  4. Gain experience with reinforcement learning algorithms

  5. Improve your paper-reading skills

  6. Improve your presentation skills

Academic Integrity

You are encouraged to work together and discuss the papers with others BUT you must write up your own paper reviews. If we find highly identical reviews without proper accreditation of collaborators, we will take action according to university policies.

Take care of yourself

Studies have shown that your mental health can benefit by maintaining a healthy lifestyle by eating well, exercising, avoiding drugs and alcohol, getting enough sleep and taking some time to relax. These practices can help you achieve your goals and cope with stress.

Even with these practices, it is natural to struggle with mental health challenges. All of us benefit from support during times of struggle. You are not alone. There are many helpful resources available on campus and an important part of the college experience is learning how to ask for help. Asking for support sooner rather than later is often helpful. If you or anyone you know experiences any academic stress, difficult life events, or feelings like anxiety or depression, we strongly encourage you to seek support. Counseling and Psychological Services (CaPS) is here to help: call 412-268-2922 and visit their website at http://www.cmu.edu/counseling/. Consider reaching out to a friend, faculty or family member you trust for help getting connected to the support that can help.

Names / identifiers

All people have the right to be addressed and referred to in accordance with their personal identity. Students are encouraged to use NameCoach to indicate the name that they prefer to be called and to identify pronouns with which they would like to be addressed. I will do my best to address and refer to all students accordingly and encourage everyone in the class to do so as well.

Inclusivity and anti-harassment policy

We must treat every individual with respect. We are diverse in many ways, and this diversity is fundamental to building and maintaining an equitable and inclusive campus community. Diversity can refer to multiple ways that we identify ourselves, including but not limited to race, color, national origin, language, sex, disability, age, sexual orientation, gender identity, religion, creed, ancestry, belief, veteran status, pregnancy, or genetic information. Each of these diverse identities shape the perspectives our students, faculty, and staff bring to our campus. We at CMU will work to promote diversity, equity and inclusion. We acknowledge our imperfections while we also fully commit to the work, inside and outside of our classrooms, of building and sustaining a campus community that increasingly embraces these core values.

Each of us is responsible for creating a safer, more inclusive environment. Unfortunately incidents of bias or discrimination do occur, whether intentional or unintentional. They contribute to creating an unwelcoming environment for individuals and groups at the university. Therefore, the university encourages anyone who experiences or observes unfair or hostile treatment on the basis of identity to speak out for justice and support, within the moment of the incident or after the incident has passed. Anyone can share these experiences using the following resources:

All reports will be documented and deliberated to determine if there should be any following actions. Regardless of incident type, the university will use all shared experiences to transform our campus climate to be more equitable and just.

The course instructors also encourage anyone who experiences or observes unfair or hostile treatment to report such incidences to the course instructors so that we can address such situations.

Accommodations for Students with Disabilities

If you have a disability and are registered with the Office of Disability Resources, I encourage you to use their online system to notify me of your accommodations and discuss your needs with me as early in the semester as possible. I will work with you to ensure that accommodations are provided as appropriate. If you suspect that you may have a disability and would benefit from accommodations but are not yet registered with the Office of Disability Resources, I encourage you to contact them at access@andrew.cmu.edu.

Accommodating diverse needs

It is my intent that students from all diverse backgrounds and perspectives be well served by this course, that students’ learning needs be addressed both in and out of class. It is my intent to present materials and activities that are respectful of diversity: gender, sexuality, disability, age, socioeconomic status, ethnicity, race, and culture. Your suggestions are encouraged and appreciated. Please let me know ways to improve the effectiveness of the course for you personally or for other students or student groups. In addition, if any of our class meetings conflict with your religious events, please let me know so that we can make arrangements for you.