Differentiable cameras and displays
Course Description
This course provides an introduction to differentiable wave propagation approaches and describes its application to cameras and displays. Specifically, the optical components of displays and cameras are treated as differentiable layers, akin to neural network layers, that can be trained jointly with the computational blocks of an imaging/display system.
PRANEETH C
Princeton University
Praneeth Chakravarthula is a research scholar at Princeton University. His research interests lie at the intersection of optics, perception, graphics , computer vision and machine learning. Praneeth's research has won several awards including two Best Paper Awards at SIGGRAPH 2022, Timothy L. Quigg student inventor of the year 2019, SIGGRAPH best in show award 2018, Best Paper Awards at ISMAR 2018, IEEE VR 2019, OSA Biophotonics 2020, SPIE Photonics West 2018 and 2020. He obtained his PhD from UNC Chapel Hill, under the advise of Prof. Henry Fuchs, on next generation everyday-use eyeglasses style near-eye displays for virtual and augmented reality.
Florian Schiffers
Northwestern University
Florian Schiffers is a Ph.D. candidate in Computer Science at Northwestern University under the supervision of Oliver Cossairt. Before joining Northwestern, he obtained master degrees in Physics and Optical Technologies from FAU Erlangen. His research interests lie in computational imaging systems that combine novel optical elements with machine learning. His current research focuses on the end-to-end optimization of computer-generated holography systems that will enable the next generation of near-eye displays.
Northwestern University, Meta
Oliver Cossairt is an Associate Professor in the Computer Science and Electrical and Computer Engineering departments at Northwestern University. Prof. Cossairt is director of the Computational Photography Laboratory at Northwestern University, whose research consists of a diverse portfolio, ranging in topics from optics/photonics, computer graphics, computer vision, machine learning and image processing. The general goal of his lab is to develop imaging hardware and algorithms that can be applied across a broad range of physical scales, from nanometer to astronomical. Prof. Cossairt has garnered funding from numerous corporate sponsorships and federal funding agencies.
Princeston University
Ethan Tseng is a Ph.D. candidate advised by Prof. Felix Heide at Princeton University and he received his B.S. in Electrical and Computer Engineering from Carnegie Mellon University. Ethan’s research focuses on optics, machine learning, and optimization for computational imaging and holographic displays. He has interned with Marc Levoy’s team at Adobe Research and in Prof. Aswin Sankaranarayanan’s Image Science Lab. Ethan’s work on nano-optics has been highlighted by Optics & Photonics News and has been featured in international media such as Vice News, BBC, NSF Discovery Files, and Jimmy Fallon’s Tonight Show.
POSTECH, Princeton University
Seung-hwan Baek is an assistant professor at POSTECH. Before joining POSTECH, he worked as a post-doctoral research associate at Princeton University and holds a Ph.D. degree in Computer Science from KAIST. His research interests lie in computer graphics and computer vision with a particular focus on computational imaging and display. His work aims to capture, model, and analyze the high-dimensional visual information of the real world originating from complex interplays between light, material appearance, and geometry. To this end, he designs end-to-end computational imaging and display systems for fundamental scientific analysis as well as diverse application domains.
Princeton University, Algolux
Felix Heide is an Assistant Professor at Princeton University and Co-Founder and Chief Technology Officer of self-driving vehicle startup Algolux. He is researching the theory and application of computational imaging and computer vision systems. Exploring imaging, vision, and display systems end-to-end, Felix's work lies at the intersection of optics, machine learning, optimization, computer graphics, and computer vision. He received his Ph.D. from the University of British Columbia. His doctoral dissertation won the Alain Fournier Ph.D. Dissertation Award and the SIGGRAPH outstanding doctoral dissertation award. He won the NSF CAREER Award 2021 and the Sony Young Faculty Award 2021.
Introduction
Welcome to the overview course of modern computational imaging and displays. Cameras and display systems are ubiquitous and pervade everyday life. However, the design and implementation of these systems still rely on outdated techniques and have not kept pace with the rapid development of modern tools, such as deep learning.
This course aims to introduce the public, from industry engineers, startup entrepreneurs, and academic researchers, to modern design techniques that have been spearheaded by the speakers of this course. We draw upon the latest state-of-the-art literature published in ACM Transactions on Graphics, Nature, and CVPR that experimentally validate the advantages of the proposed techniques. Through this course, the audience will achieve a concrete understanding of these modern design techniques and how to implement these methods in their own applications.
This course will focus on four core topics: differentiable wave propagation, building practical differentiable displays, macro to nano camera design, differentiable illumination and sensing. These topics are described in further detail below.
Course Overview
Praneeth Chakravarthula (Princeton)
Florian Schiffers and Oliver Cossairt (Northwestern, Meta)
Felix Heide (Princeton)
Ethan Tseng (Princeton)
Seung-Hwan Baek (Princeton, POSTECH)
Prerequisites
This is an introductory level course and has no prerequisites. Academic researchers with imaging, vision, or optics background all alike will be able to attend the course without prerequisites. Similarly, the course is designed to include practitioners as attendees in the camera, imaging, optics, visual effects, robotics, or autonomous vehicle space. Practical coding examples will facilitate an engaging course.
Code
Jupyter Notebook ( Google Colab):
GitHub
https://github.com/facebookresearch/holotorch
Neural Nano-Optics
Wave propagation and digital holography
Praneeth Chakravarthula
The speaker will introduce techniques for modeling physical light wave propagation, specifically covering near-field angular spectrum, fresnel propagation, and far-field Fourier propagation. The speaker will describe how backpropagation through these physical models can be enabled by Wirtinger gradients. With these differentiable models, he will describe state-of-the-art CGH methods. Afterward, the speaker will introduce the relationship between light fields and these wave models and show how 3D holograms can be generated with accurate parallax and occlusion. The MATLAB/PyTorch code for the implementation and optimization of these holograms will be provided and described.
The technical contents for this section of the course are described in more detail below.
Modeling Wave Physics This module introduces the audience to the basics of light wave propagation and describes the scalar diffraction integral and tractable fast Fourier transform-based methods, including angular spectrum propagation, near-field Fresnel propagation, and Far-field Fourier propagation based on complex math. The audience will also be introduced to the various sampling constraints, trade-offs, accuracy assumptions and validity regimes. At the end of this module, the attendees will be able to understand wave propagation scalar diffraction integral in general and when to apply the several propagation models. In the end, the audience will also be introduced to holography and computer-generated holography (CGH) methods.
Differentiability via Wirtinger Gradients The introduced wave propagation models defined in the complex domain are not differentiable and hence are difficult to be applied in combination with gradient descent optimization methods. We introduce the attendees to Wirtinger calculus, specifically to Wirtinger gradients, used to remedy this problem and pose the CGH task as a differentiable first order optimization problem. Moreover, the attendees will also be introduced to the implementation of Wirtinger gradients in PyTorch and TensorFlow to use with their Stochastic Gradient Descent (SGD) optimization methods.
Light Fields and 3D Holography The audience will be introduced to light fields and their relationship to 3D holograms. Approaches to converting a light field into a hologram and vice versa will be described. Specifically, the windowed Fourier transform (WFT), hogels (holographic elements), and holographic stereograms will be discussed. The attendees will also be introduced to the implementation of WFT, and the corresponding code for generating holograms from light fields will be covered. Additionally, advantages of light fields over alternative scene representations such as RGB-D, focal stacks, multi-plane will be discussed in detail.
holotorch - differentiable coherent light transport in pytorch
Oliver Cossairt, Florian Schiffers
In this section, we present HoloTorch; a PyTorch library tailored for prototyping Fourier-Optics-based setups and algorithms for Computer Generated Holography. At its core, HoloTorchis a differentiable, coherent light-transport renderer capable of simulating, optimizing, and calibrating holographic displays. HoloTorch allows to quickly assemble a coherent optical system in simulation and use its pre-implemented optimization routines for end-to-end calibration approaches such as Neural Holography in both simulation and experiment.
This tutorial will go through the basics of HoloTorch and implement several holographic algorithms: We first show how to assemble a four-f-system to simulate holograms encoded with Double-Phase-Encoding. Second, we demonstrate to quickly optimize for SLM-patterns in a near-field holographic setup and an etendue expansion configuration based on a thin scatter. Last, we show how HoloTorch can optimize for holographic Neural Etendue Expansion, where SLM patterns and optical elements are optimized simultaneously from a large dataset of target images.
The Differentiable Camera
Felix Heide
Neural Proxies for Camera ISPs We demonstrate how neural networks can model complex black-box image signal processors (ISP), including both hardware ISPs for real-time automotive applications and commercial photo-finishing packages. The neural models will enable gradients to flow from the task objective back to the configurable parameters. This module will especially be of interest to engineers working on hardware ISPs where settings such as exposure, white-balance, gamma, are difficult to configure by hand. It will also be of interest to artists and photographers working with photo-finishing pipelines where slider settings such as saturation and sharpening are often adjusted manually. The audience will obtain an understanding of how to integrate these neural proxies into their own applications.
Integration with Commercial Optics Design Software The speaker will show how to model optical simulations that are conventionally done by commercial software suites such as Zemax/Code V. The design strategy for building neural proxies for optical simulation, including spatially varying aberrations encoded by point spread functions (PSF) will be presented. The audience will learn how to describe the parameters of an optical stack, such as lens curvature, lens distance, and conic constants, as inputs into a fully differentiable neural network that simulates PSFs. The attendees will learn how to use these optical proxies to design compound optical hardware, and how to integrate these proxies into task-specific applications such as object detection. Furthermore, the audience will obtain an appreciation for designing optics by using task-specific objectives instead of intermediary merit functions such as spot size and wavefront error.
Neural Nano-Optics for High-quality Thin Lens Imaging
Ethan Tseng
We will introduce entirely new optical systems that can miniaturize bulky cameras to the size of a salt grain. To this end, we will introduce approaches for by passing computationally and optically prohibitive barriers to this effort. The audience will learn about rigorous coupled wave analysis (RCWA) and fast methods for approximating RCWA while maintaining end-to-end differentiability. The audience will also learn about state-of-the-art deconvolution approaches that combine traditional filters with modern machine learning, and how to integrate these methods together with the metasurface simulation block.
Differential Illumination and Temporal Sensing
Seung-Hwan Baek
The last section of this course will unify the projection and imaging techniques that have been taught throughout this course. The speaker will describe co-design of illumination and sensing with 3D-imaging applications including active-stereo imaging and time-of-flight imaging. In this section, the speaker will describe differentiable ray-based and wave-based light transport and how this can be applied to temporal and depth sensing, potentially enabling entirely new imaging modalities.
Hybrid Ray-based and Wave-based Light Transport for Active Stereo The speaker will cover the difference of the wave- and ray-based propagation models. This is followed by demonstrating the application of utilizing a wave-based propagation model for simulating the projection of laser illumination patterns to a scene. The speaker will then introduce a hybrid model that combines the wave- and ray-based models for illumination and sensing respectively in order to build a learned active-stereo system for 3D imaging. The speaker will show how to design a diffractive optical element (DOE) using the light transport models covered throughout this course. The speaker will also revisit the different conventional iterative Fourier Transform Algorithms (IFTA) and the differentiable end-to-end optimization methods taught throughout the course. Code will be provided to the audience and covered in detail.
Differentiable Time-of-flight Imaging with Optimized Micro-lens Apertures The speaker will cover the basics of indirect time-of-flight (ToF) imaging including its optics, electronics, and reconstruction process. Then the attendees will be introduced to ways in which a differentiable light simulation can be incorporated into a ToF-imaging system and improve the sensing by removing "flying pixels". Specifically, the speaker will discuss differentiable formulation of the sensing model and joint