Differentiable and Probabilistic Programming Languages

"Everything changes and nothing stands still." — Heraclitus

"Chance governs all things; necessity, which is far from having the same purity, comes only later." — Luis Buñuel

This class is about taking derivatives of programs (automatic differentiation) and interpreting programs as probabilistic models: change and chance.  We will take a broad and eclectic view on these subjects: what does it mean to take derivatives of discrete programs?  How do you conditionally sample from a program?  Is the derivative of a computable function always computable?  How can you verify a randomized program?

We will also cover a range of interesting applications including physical simulation, rendering, 3d reconstruction problems, cognitive science models, inverse design problems, and more!

The first half of the course will cover the basic ideas and implementation of automatic differentiation of programs and probabilistic inference over programs.  The second half of the course will function like a paper seminar.  Students will complete 3 assignments covering basic techniques, followed by a group project of their choosing.

 Course Information

Expectations

Students are expected to attend lecture and ask questions.  The coursework will consist of 3 programming assignments (done in groups) in the first half of the quarter, and a group project in the second half of the quarter.  Students should expect to present one paper during the second half of the quarter.

We strongly suggest attempting to spend an hour each day making slow and steady progress.  Procrastinating until deadlines will make the projects very unpleasant.  In general, you should assume that it may take up to 24 hours for course staff to respond to questions, and that course staff will not be available on weekends or holidays.  This is another important reason to start on the projects early.

Groups & Class Research Projects

Students will work in groups in this class.  A group consists of 2-3 students.  Your group will be most successful if the members have complimentary skills.  Ideally, every group will have at least one member who is comfortable with the continuous math and applications and at least one member who is comfortable with programming language implementation.  In order to ensure everyone is set up for success, you will be required to get instructor approval for your group.

You will spend the first 6 weeks doing class assignments, but to do a good job, you will have to start planning for your class research project as soon as possible.  Your group will have periodic meetings with the instructor to help shape and select a good project.

 Prerequisites

You will find this class easiest if you feel very comfortable with compiler implementation and continuous math.  Since this describes very few students, (perhaps an infinitesimal number; a set of measure zero?) we will not assume such background.  The class will start with a module on programming language implementation built around the assignment code base, and review the relevant mathematics as we progress.

Grading and Late Policy

The course grade will be composed of 3 assignments (3/6 of grade), a class project (2/6), and leading one class paper discussion (1/6).  The first assignment will be focused on familiarizing students with the compiler they will modify in subsequent assignments.  Each student is allowed up to 2 late days per assignment, but no more.  Exceptions to this policy will only be given in extreme circumstances with prior instructor approval.

Discussion is allowed between groups, but your group must write and turn in your own work in its entirety.  Please use the discussion board to ask and answer questions, but do not post your source code.

Students who regularly participate in class and turn in projects on time will be eligible to make up missed points on projects by demonstrating comprehension of the relevant concepts through quizzes or other means at the discretion of the instructor.  (Basically, if you're putting in your best effort and can eventually prove you learned everything this class is supposed to teach you, then you'll get a good grade)

AI Policy

I don't want to worry about this, but unfortunately we have to navigate this issue one way or another.  Here are some rules I've come up with:

Resources and Textbooks