Class time: Monday and Wednesday, 12:20 pm - 2:05 pm

Classroom: STH B22 (745 Commonwealth Ave)

Course website: https://sites.google.com/view/bu-ec-500-au26-chao 

Instructor: Prof. Wei-Lun (Harry) Chao

Email: chao209@bu.edu 

Office hours: Wednesday 2:10 pm - 3 pm (PHO437) or by appointment

TA: Sanjana Sanjeev Kumar 

Email: sanjask@bu.edu

Office hours: TBA (TBA) 

Syllabus: link

Course Description:  

This course focuses on the foundations of computer vision, with particular emphasis on learning-based methods and 3D. To build background, the course covers the basics of image formation, camera modeling, machine learning, and neural networks. With this groundwork, the course introduces image-processing-based methods and probabilistic models of images. Then, the course explores modern neural network architectures for computer vision, including convolutional neural networks and transformers. The course then builds upon these models to develop algorithms for image feature extraction, visual recognition, image generation, and vision-and-language understanding. Moving beyond single images, the course further introduces stereo vision and multi-view vision, including structure from motion and neural radiance fields. Finally, the course introduces algorithms for motion estimation and tracking. Along with the course, representative applications for computer vision will be introduced and discussed.

The course will consist of two lectures per week, each lasting 1 hour and 45 minutes (including a break).

Most lectures will use slides. Python coding may be incorporated into the lectures. Students can expect bi-weekly homework assignments that will take them 5-15 hours to complete. 

Course Credits: 4 units

Course Goals / Objectives: 

• Master fundamental and recent computer vision concepts and algorithms

• Be competent with computer vision application design and evaluation

• Gain a deep understanding of learning-based algorithms and 3D inference for computer vision

• Be exposed to original research and applications in computer vision

• More broadly, the aim is to provide students with a strong foundational background, enabling them to pursue computer-vision-centered or machine-learning-centered PhD paths or explore future opportunities in the computer vision, machine learning, and artificial intelligence industries.

Course Topics: 

1. Introduction to computer vision

   a. Introduction to the course

   b. A simple vision system

2. Image formation

   a. Concepts of imaging and lenses

   b. Images and 3D geometry

   c. Camera modeling

   d. Cameras as linear systems

3. Foundations of image processing

   a. Linear filtering and convolution

   b. Fourier analysis

   c. Blur filters, image derivatives, and filter banks

   d. (Up/down) sampling

   e. Image pyramids

4. Foundations of learning

   a. Introduction to learning

   b. Gradient-based learning algorithms

   c. Generalization

   d. Neural networks as distribution transformers

5. Probabilistic models of images

   a. Color

   b. Statistical image models

   c. Textures

6. Neural architectures for vision

   a. Convolutional neural nets

   b. Transformers

7. Generative image models and representation learning

   a. Representation learning

   b. Generative models

8. Understanding vision with semantics and language

   a. Visual recognition

   b. Vision and language

9. Challenges in learning-based vision

   a. Data bias and shift

   b. Robustness and generality

   c. Transfer learning and adaptation

10. Understanding geometry

   a. Stereo vision

   b. Homographies

   c. Depth estimation from single images

   d. Feature detection and matching

   e. Multi-view geometry and structure from motion

   f. Radiance fields

11. Understanding motion

   a. Motion estimation

   b. Optical flow estimation

   c. Object tracking

Expected Outcome:

1) Understand how images are formed, and their relationship to the 3D world through camera modeling

2) Understand the basic concepts and algorithms for inferring 3D from 2D images

3) Implement a simple vision system to infer 3D from a single image

4) Understand the foundations of image processing, including filters, convolutions, Fourier analysis, and sampling

5) Implement and test 2D Fourier analysis, convolution, simple linear filters, and sampling

6) Understand the foundations of machine learning, including basic algorithms, challenges, and neural networks

7) Understand how images are represented and generated in computational models

8) Understand how to recognize semantic concepts from images

9) Implement and test vision recognition, representation learning, or generative models using deep learning

10) Understand how to infer 3D geometry and motion from single and multiple images

11) Implement and test geometry processing and 3D inference algorithms

12) Apply gained knowledge to a real-life computer vision problem through a team project

13) Deliver a spotlight presentation in class and write a scientific paper–style project report

14) Be familiar with the Python/PyTorch programming environment 

Undergraduate prerequisites: 

• ENG EK 130 (Python)

• ENG EK 103 (linear algebra) 

• ENG EK 381 (probability & statistics) 

• ENG EC 414 (machine learning)

• ENG EC 401 (signals and systems)

• Or consent of the instructor

Graduate prerequisites: No prerequisites, but prior attendance of the following courses is a plus

• ENG EC 503 (learning from data)

• ENG EC 520 (digital image processing)

• ENG EC 523 (deep learning) 

Other suggested background:

• Students are expected to have a decent degree of mathematical sophistication, including linear algebra, multivariate calculus, probability, and statistics. Students are also expected to know programming, algorithm design, and data structures.

• Programming in Python 3 is required. Programming in PyTorch and using Hugging Face might be needed.

• Review materials can be found here: linear algebra, probability, Python-1, Python-2, Python-3 

• Also, check HERE for a set of slide decks for linear algebra

Announcements, communications, and discussions: 

• We will make normal announcements using Blackboard. Announcements of urgent matters will be emailed to your @bu.edu address. If you do not regularly read that account, make sure you forward it to somewhere that does.

• We will use Piazza for discussions. If you have questions about the course materials or policy, please post them on Piazza. The TA and I will also monitor these discussions and answer as appropriate, but students should be active and feel free to use the forums to have group discussions as well.

• Please only use email to contact the instructor or the TA for urgent or personal issues. Any e-mails sent to the instructor or TA should include the tag "[BU-EC-500]" in the subject line. (This ensures we can filter and prioritize your messages.) We reserve the right to forward any questions (and their answers) to the entire class if they should prove relevant. Please indicate if you wish to be anonymized (i.e., have your name removed) in this case.

Reading before or after the class plays an important part in your learning. Readings will be assigned for each lecture. Scientific papers, book chapters, and technical material may be suggested in class and provided by the instructor. 

Required Textbook: 

• Antonio Torralba, Phillip Isola, and William T. Freeman, Foundations of Computer Vision. MIT Press, 2024. (Ebook available through https://visionbook.mit.edu/) - There will be a reading assignment for each lecture.

Suggested References: 

• Richard Szeliski, Computer Vision: Algorithms and Applications (second edition). Springer, 2022.

• David Foster, Generative Deep Learning: Teaching Machines To Paint, Write, Compose, and Play (second edition). O'REILLY, 2023. 

Other Good References:

• Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola, Dive into Deep Learning. 2021. https://d2l.ai/index.html

• Simon J. D. Prince, Understanding Deep Learning. The MIT Press, 2023. https://udlbook.github.io/udlbook/ 

• Christopher M. Bishop and Hugh Bishop, Deep Learning: Foundations and Concepts. Springer, 2024. 

Other Good CV Courses:

• Stanford CV: http://vision.stanford.edu/teaching/cs131_fall2223/ and https://cs231n.stanford.edu/

• MIT CV: http://6.869.csail.mit.edu/sp22/schedule.html and https://advances-in-vision.github.io/schedule.html

• CMU CV: http://16385.courses.cs.cmu.edu/spring2024/

• Brown CV: https://browncsci1430.github.io/

• NYU CV: https://www.sainingxie.com/cv-fall2024/ 

• Wisconsin-Madison CV: https://sites.google.com/view/cs639spring2023dlcv

• Michigan CV: https://web.eecs.umich.edu/~justincj/teaching/eecs442/WI2021/ and https://web.eecs.umich.edu/~justincj/teaching/eecs498/WI2022/

• Cornell CV: https://www.cs.cornell.edu/courses/cs4670/2021sp/ and https://www.cs.cornell.edu/courses/cs6670/2023fa/

PyTorch:

• https://pytorch.org/

• https://pytorch.org/tutorials/ 

• https://pytorch.org/vision/stable/index.html

Useful Reference:

• Kaare Brandt Petersen and Michael Syskind Pedersen, The Matrix Cookbook

Homework: 

• There will be around 6 homework assignments.  

• Each assignment may include a problem set and a programming set. 

• Programming in Python 3 is required. 

• Blackboard (and other platforms, such as Gradescope and GitHub) will be used for submissions. 

• For the problem set and the report of the programming set, we will only allow PDF submission.

• You must strictly follow the homework and submission instructions. 

Quizzes: 

• In-class quizzes will be paper-based, based on the materials from the previous lecture and the last homework (after its deadline). 

• There will be around 20 quizzes. The 16 highest scores will be counted.

• The linear algebra quiz (on 9/14/2026) will be in-class and paper-based as well.

Midterm exam: 

• The midterm is in person and closed-book. 

• Exam materials/questions may come from the reading listed in the schedule below.

Final project:

• Team project involving algorithm development; proposal, milestones, report(s), and in-class presentation(s). 

• Strongly suggest that you get familiar with PyTorch, GitHub, and Hugging Face ASAP.

• Details will be announced in class and can be found on Blackboard.

Quiz:

• Quizzes will take place at the beginning of each lecture. 

• Closed book and no cheat sheets.

• Cheat sheets are allowed for the linear algebra quiz: two letter-size cheat sheets (4 pages).

Homework:

• Late submission policy: Late submissions within 1 day will be deducted by 20%. Late submissions within 1-3 days will be deducted by 40%. Late submissions after 3 days will receive 0 grade.

• No late submissions are accepted after solutions are released (specifically for Homework 3, which is due before the midterm exam). 

• Homework should be neat and professional and follow the required format. In particular, homework on torn sheets, scrap paper, or not well-scanned into a single file will not be accepted.

• Homework is to be done individually. Of course, the discussion between students is allowed and encouraged, but the actual homework should be completed separately. You have to list with whom you discussed.

• Questions about homework or exams should be made in a timely fashion. Any complaints about homework grading must be made within 1 week of when the item is returned or before 12/16/2026 (whichever comes first). Do not wait until the end of the semester!

Exam:

• The midterm is in person and closed-book. 

• Three letter-size cheat sheets (6 pages) are allowed.

• Excuses from scheduled exams can be accepted only in case of academic-relevant travel, personal sickness requiring medical care, or severe accidents in the immediate family (documentation required).

• Academic-relevant travel must be reported one month before the exam. 

Homework-0 (self-check): PDF

Slide deck 1: PDF, PPT

Future course slides will be on Blackboard