Welcome to Spring 2022

ESE 650: Learning in Robotics

Term: Spring 2022

Location: ANNS 110

Time: Tuesday & Thursday, 1:45 PM - 3:15 PM

Course Instructors


Office Hours: Tue, 3:30-5pm



Office Hours: Mon, 1-2:30pm, link (starting Feb 28)

Course Description

A robot is a machine that senses its environment using sensors, interacts with this environment using actuators to perform a given task and does so efficiently using previous experience of performing similar tasks. In ESE 650, we will cover the fundamentals of these three aspects of robotics: perception, planning, and learning. A tentative list of topics includes state estimation (EKF, UKF, Particle Filters, visual-inertial odometry), control and planning (LQR, MDPs, sampling-based planning), reinforcement learning (policy gradients, Q-learning, Imitation Learning, Offline RL) and some miscellaneous topics (meta-learning and formal verification). The coursework will have both applied and theoretical aspects. This is a graduate level class; undergraduates will need permission from the instructor to enroll. Some experience with, or appreciation of, robotics is recommended.

Course Syllabus

Module 1: State Estimation

Background on probability

Topics: Markov chains, Hidden Markov Models, Kalman Filter, Extended and Unscented Kalman Filter, particle filters, occupancy grids

Module 2: Control and Planning

Background on linear control, dynamic programming

Topics: Markov Decision Processes, Value and Policy Iteration, Bellman equation, Linear Quadratic Regulator, Linear Quadratic Gaussian, sampling-based motion planning

Module 3: Reinforcement Learning

Background on deep learning and optimization

Topics: Imitation Learning, Policy gradient, Q-Learning, Inverse RL, Model-based RL, Offline RL



  1. Proficiency in programming (ENGR 105, CIS 110, CIS 120 or equivalent). All assignments will be based on Python but if you have used a similar language like MATLAB before, you should be able to pick up Python easily. Recitation sessions will provide preparatory material.

  2. Probability (ESE 301, STAT 430, CIS 261, ENM 503, ESE 530 or equivalent)

  3. Linear Algebra (Math 312, EAS 205 or equivalent)


  1. Machine Learning or Data Analysis (ESE 305, ESE 402, ESE 545, CIS 519/520, or a first machine learning course)

  2. (Soft recommendation) Optimization (ESE 304, ESE 504, ESE 605)

Textbook References:

Detailed instructor notes in addition to supplementary reading material from the following books.

  1. (Thrun) “Probabilistic Robotics” by Sebastian Thrun, Wolfram Burgard and Dieter Fox. PDF

  2. (Barfoot) “State Estimation for Robotics” by Tim Barfoot. PDF

  3. (Lavalle) “Planning Algorithms” by Steve Lavalle. PDF

  4. (Sutton) “Reinforcement Learning: An Introduction” by Richard Sutton and Andrew Barto PDF

  5. (d2l) Dive into Deep Learning by Aston Zhang, Zack Lipton, Mu Li and Alex Smola available at https://d2l.ai is a good reference to read about deep learning.

  6. (Russell) “Artificial Intelligence: A Modern Approach” by Stuart Russell and Peter Norvig. PDF

The following books contain some advanced material. You can use it for your own reference and to brush up fundamentals of machine learning and optimization.

    1. “Pattern Recognition and Machine Learning” by Christopher Bishop. PDF

    2. “An Invitation to 3-D Vision: From Images to Models“ by Yi Ma, Stefano Soatto, Jana Kosecka, Shankar Sastry. PDF

    3. “Reinforcement Learning and Optimal Control” by Dmitri Bertsekas. Material

    4. “Feedback Systems: An Introduction for Scientists and Engineers” by Karl Johan Astrom and Richard M. Murray, PDF

    5. (Advanced) “Linear Systems Theory” by João P. Hespanha. Website

    6. (Fairly advanced) “Stochastic Models, Information Theory, and Lie Groups, Volume 1: Classical Results and Geometric Methods” by Gregory Chirikjian. PDF

Grading Policy:

  • 4 homeworks (60% in total)

  • Mid-term exam (20%)

  • Final project (teams of 3) (20%)

Each student will have 10 “late days” to use during the semester which can be used to submit the deliverables later than they are due. Deliverables that are submitted late after exhausting this quota will result in 50% credit deduction per day. Late days cannot be used for the final project submission.

Canvas: https://canvas.upenn.edu/courses/1626578

Ed: edstem.org/us/courses/16341/discussion/

Lecture Notes (2021): https://pratikac.github.io/pub/21_ese650.pdf

Teaching Assistants

Chris Hsu

Email: chsu8@seas.upenn.edu

Office Hour: Wed, 9:00-10:30pm

Link: Zoom

Jason Ma

Email: jasonyma@seas.upenn.edu

Office Hour: Thu, 12pm-1:30pm


Arjun Nanda

Email: manvn@seas.upenn.edu

Office Hour: Fri, 11-12:30pm


Chuning Zhu

Email: zchuning@seas.upenn.edu

Office Hour: Mon, 4-5:30pm

Link: Zoom

Kun Huang

Email: huangkun@seas.upenn.edu

Office Hour: Thur, 3:30-5pm

Link: Zoom

Agelos Kratimentos

Email: agelosk@seas.upenn.edu

Office Hour: Tues, 5-6:30pm

Link: Zoom

Code of Academic Integrity

University of Pennsylvania's CIS/ESE department encourages collaboration among graduate students. However, it is important to recognize the distinction between collaboration and cheating, which is prohibited and carries serious consequences. Cheating may be defined as using or attempting to use unauthorized assistance, material, or study aids in academic work or examinations. Some examples of cheating are: collaborating on a take-home exam or homework unless explicitly allowed; copying homework; handing in someone else's work as your own; and plagiarism. Any student suspected of cheating will be reported to the Office of Student Conduct.

You are encouraged to collaborate with your peers for solving problems in the homework, reading books and curating other instructional materials to improve your understanding of the concepts taught in the class. While doing so, you might generate code/pseudo-code/solutions for the homeworks/project. When you begin to write your submission you should keep aside all these materials (including your friends) and do things “from scratch”. In short, everything you write/code and submit should be your own work done independently.

You should disclose all collaborations in your submission at the top. If you came across some code as a part of your homework/project you must mention it.

Collaboration is different from cheating. The latter will have serious consequences. Cheating is defined as attempting, abetting or using unauthorized assistance (knowledgeable friend who is not taking the class) or material (e.g., online code). Some examples of cheating are: copying someone else’s work for homework/exams, handing in someone else's work as your own or handing in stuff from the Internet as your own work. These will not be tolerated. Your score for that particular homework or exam will be zeroed out if found guilty, you will be penalized one letter grade and this incident will be reported to the university.

Inclusive Environment

If you have any special needs that affect your coursework/attendance, please talk to the instructor and I will do my best to accommodate them. We are all adults here and we should together create an open and inclusive academic environment. This includes being respectful while interacting with your peers, being receptive to their ideas/thoughts, and, in general, maintaining a scholarly discourse both inside and outside the classroom. Transgressions will be reported to the university.