16-881: Deep Reinforcement Learning for Robotics

Course Info:

Days TR

Room NSH 3002

Time 4:30 - 5:50 PM

Instructor David Held (Office hours upon request)


      • Brian Okorn (Office hours: Wednesdays 3-4 PM, NSH 4222)
      • Ben Newman (Office hours: Thursdays 3:15 - 4:15 PM, NSH 4506)

Instructor+TA Email List 16881-tas@lists.andrew.cmu.edu

Course Discussion / Announcements: https://piazza.com/cmu/spring2019/16881 (make sure to sign up to receive announcements and to submit assignments)

Course Description

Each class, 2 papers will be presented; one paper will be a paper on deep RL; the other will be a paper on robotics, which will have an impressive robotics result, possibly using RL but not deep RL. The class will read and discuss these papers and try to understand:

  • How did the robotics paper achieve its result without deep RL?
  • What are the strengths and limitations of the approach described in the robotics paper?
  • What insights can we take away from this paper?
  • What are the strengths and limitations of the method described in the deep RL paper?
  • How can the method described in each paper be improved?

Students will also work on a class project related to RL of their choosing.

Class format

Each class 2 papers will be presented:

  • 1 robotics paper (no deep RL)
  • 1 deep RL paper

We will discuss these papers, try to understand them, and draw insights:

  • How does the robotics paper achieve its results without deep RL?
  • What new advantages does the deep RL paper bring?
  • What new methods / ideas does this comparison inspire?

Class timing

The timing for the class will be as follows:

  • 23 minutes for presentation 1 (+2 minutes for clarification questions and changeover)
  • 23 minutes for presentation 2 (+2 minutes for clarification questions and changeover)
  • 10 minute small group discussion
  • 20 minute class discussion


Students are expected to have already have a basic understanding of reinforcement learning, such as from 10-703, 16- 748, 16-831, or a similar course, prior to taking this course. If you need a refesher, I highly recommend Reinforcement Learning: An Introduction by Richard Sutton and Andrew Barto, especially Chapters 2-6 and 13. Also, the first 2 lectures of the course will cover a quick review of reinforcement learning.


  • Presentations: 30%
  • Paper reviews: 40%
  • Class project: 30%

Educational Outcomes

  1. Become familiar with some classic robotics papers and their approaches
  2. Become familiar with some recent deep reinforcement learning papers
  3. Think about the tradeoffs between different approaches for solving a robotics problem
  4. Gain experience with reinforcement learning algorithms
  5. Improve your paper-reading skills
  6. Improve your presentation skills

Academic Integrity

You are encouraged to work together BUT you must write up your own paper reviews. If we find highly identical reviews without proper accreditation of collaborators, we will take action according to university policies.

Take care of yourself

Do your best to maintain a healthy lifestyle this semester by eating well, exercising, avoiding drugs and alcohol, getting enough sleep and taking some time to relax. This will help you achieve your goals and cope with stress. All of us benefit from support during times of struggle. You are not alone. There are many helpful resources available on campus and an important part of the college experience is learning how to ask for help. Asking for support sooner rather than later is often helpful. If you or anyone you know experiences any academic stress, difficult life events, or feelings like anxiety or depression, we strongly encourage you to seek support. Counseling and Psychological Services (CaPS) is here to help: call 412-268-2922 and visit their website at http://www.cmu.edu/counseling/. Consider reaching out to a friend, faculty or family member you trust for help getting connected to the support that can help.