"Everything changes and nothing stands still." — Heraclitus
"Chance governs all things; necessity, which is far from having the same purity, comes only later." — Luis Buñuel
This class is about taking derivatives of programs (automatic differentiation) and interpreting programs as probabilistic models: change and chance. We will take a broad and eclectic view on these subjects: what does it mean to take derivatives of discrete programs? How do you conditionally sample from a program? Is the derivative of a computable function always computable? How can you verify a randomized program?
We will also cover a range of interesting applications including physical simulation, rendering, 3d reconstruction problems, cognitive science models, inverse design problems, and more!
The first half of the course will cover the basic ideas and implementation of automatic differentiation of programs and probabilistic inference over programs. The second half of the course will function like a paper seminar. Students will complete 3 assignments covering basic techniques, followed by a group project of their choosing.
Instructor: Gilbert Bernstein (gilbo@cs.washington.edu)
Office Hours: Wednesday 3pm at CSE2 235
TA: Sirui Lu
Office Hours: Thursday 2-3pm at CSE2 150
Lecture: WF 1:30-3pm in CSE2 271
Students are expected to attend lecture and ask questions. The coursework will consist of 3 programming assignments (done in groups) in the first half of the quarter, and a group project in the second half of the quarter. Students should expect to present one paper during the second half of the quarter.
We strongly suggest attempting to spend an hour each day making slow and steady progress. Procrastinating until deadlines will make the projects very unpleasant. In general, you should assume that it may take up to 24 hours for course staff to respond to questions, and that course staff will not be available on weekends or holidays. This is another important reason to start on the projects early.
Students will work in groups in this class. A group consists of 2-3 students. Your group will be most successful if the members have complimentary skills. Ideally, every group will have at least one member who is comfortable with the continuous math and applications and at least one member who is comfortable with programming language implementation. In order to ensure everyone is set up for success, you will be required to get instructor approval for your group.
You will spend the first 6 weeks doing class assignments, but to do a good job, you will have to start planning for your class research project as soon as possible. Your group will have periodic meetings with the instructor to help shape and select a good project.
Basic Calculus, Linear Algebra and Probability
Python Programming (for assignments)
You will find this class easiest if you feel very comfortable with compiler implementation and continuous math. Since this describes very few students, (perhaps an infinitesimal number; a set of measure zero?) we will not assume such background. The class will start with a module on programming language implementation built around the assignment code base, and review the relevant mathematics as we progress.
The course grade will be composed of 3 assignments (3/6 of grade), a class project (2/6), and leading one class paper discussion (1/6). The first assignment will be focused on familiarizing students with the compiler they will modify in subsequent assignments. Each student is allowed up to 2 late days per assignment, but no more. Exceptions to this policy will only be given in extreme circumstances with prior instructor approval.
Discussion is allowed between groups, but your group must write and turn in your own work in its entirety. Please use the discussion board to ask and answer questions, but do not post your source code.
Students who regularly participate in class and turn in projects on time will be eligible to make up missed points on projects by demonstrating comprehension of the relevant concepts through quizzes or other means at the discretion of the instructor. (Basically, if you're putting in your best effort and can eventually prove you learned everything this class is supposed to teach you, then you'll get a good grade)
I don't want to worry about this, but unfortunately we have to navigate this issue one way or another. Here are some rules I've come up with:
If you use Large Language Models (LLMs, e.g. ChatGPT or Co-pilot) or other forms of generative AI for an assignment, please describe how they were used. This will help me and other instructors refine our policies.
By default, I will treat unattributed use of LLMs (likewise other generative AI) to generate code, text or images as a form of plagiarism. I realize this is an open issue. However, I have to set some policy. In general, using code from any external source without attribution has always been a form of plagiarism. Cite your sources.
Using LLMs analogously to how you would use an internet search engine is definitely OK. (e.g. asking questions to learn about topics, or when you otherwise might look something up on Stack Overflow or Github) (note: copying code from Stack Overflow or Github without attribution is also a form of plagiarism)
DO NOT upload class assignments or other sensitive material to an LLM or generative AI service, including Co-pilot. There are 2 reasons for this:
1. Entering problem set questions or code into a chatbot or similar tool is roughly the same thing as posting the question on Stack Overflow and asking other people to do your homework.
2. I do not want assignments to be incorporated into the training data corpus of LLMs (or other models). Experiments with LLMs indicate that they are surprisingly good at reproducing unique text from specific sources such as personal websites. Uploading (intentionally or not) partially complete assignments could easily lead to the same issues as if you were to share or post those partially complete assignments with your current classmates or with students in future years. I have put a lot of time into designing the assignments; please do not ruin them for others.
Types and Programming Languages by Benjamin Pierce is the standard textbook for a graduate level Programming Languages class. It covers many things I don't cover, and does not cover everything I cover, but is one of the truly great textbooks in Computer Science.
Evaluating Derivatives by Andreas Griewank and Andrea Walther is the classic and authoritative textbook on Automatic Differentiation.
Introduction to Probabilistic Programming by Jan-Willem van de Meent, Brooks Paige, Hongseok Yang, and Frank Wood is an attempt at a comprehensive textbook on the subject.
The Design and Implementation of Probabilistic Programming Languages by Noah Goodman and Andreas Stuhlmüller is more approachable but more restricted in scope.