Meets: Tue 1.30-4.30 p.m ET

Instructor: Dinesh Jayaraman (dineshj [at] seas.upenn.edu)

Meeting link will be communicated over email if you are already in the class are on the waitlist.

If you are on the waitlist, attend the first class and speak with me afterwards.

Recommended Background

• Graduate-level machine learning (CIS 419/519 or equivalent)

• Undergraduate level computer vision

• Undergraduate level robotics

• Familiarity with at least one deep learning framework, pref. Pytorch

Most importantly, you must be able to understand and analyze conference papers in these areas. Talk to me if you are unsure if the course is a good match for your background. I would recommend scanning through a few papers on the syllabus to gauge what kind of background is expected. You don't have to be familiar with every single algorithm or tool a given paper mentions, but you should feel comfortable following the key ideas.

Topics

This is a graduate seminar course in computer vision and machine learning applied to robotic control. Perception and control have long been studied disjointly across many disciplines: computer vision, robotics, control theory, reinforcement learning, and cognitive science. Through presentations and discussions of recent academic papers from across disciplines, interspersed with lectures, this course will aim to synthesize a common understanding of recent advances in data-driven methods to close the robotic perception-action loop, to actively analyze the strengths and weaknesses of current approaches, and to identify interesting open questions and possible directions for future research. See below for an outline of the topics.

The majority of the course will consist of student presentations, experiments, and paper discussions. For a lecture-based course covering some of these topics, check out the ESE 650 offering by Pratik Chaudhari and C. J. Taylor.

Requirements and Grading Summary

• Paper presentations (Individual. 2 times.): 20%

• Experiment presentation (Random team of 3. 3 times): 20%

• Class and Piazza discussion participation: 15%

• Weekly paper reviews (Random team of 2. 7 weeks x 2 papers each week): 20%

• Final project (Form teams of 2 or 3!): 25%

Syllabus overview

• Vision for Robots

  • Mid-Level Visual State Estimation

  • Direct Perception

  • Active and Interactive Perception

• Learning-Based Control

  • Predictive Models and Forward Dynamics Models

  • Model-Based Reinforcement Learning and Visual Servoing

  • Model-Free Reinforcement Learning and Sim-to-Real Transfer

  • Learning from Demonstrations

• Self-Supervised Image Representations

  • Unstructured Full-Scene Representations

  • Object and Keypoint-Structured Representations

• Advanced Topics

  • Contact-Rich Manipulation and Tactile Sensing

  • Miscellaneous Topics

Acknowledgements

This course format is modeled after my PhD advisor Kristen Grauman's wonderful CS381V course at UT Austin.