Talks

Deep priors

Andrea Vedaldi, 13:05-13:35 UTC+1

https://youtu.be/tjxII31RegQ?t=387

Deep convolutional neural networks can learn extremely effectively from complex visual data. In this talk, I will suggest that part of this ability can be explained by two learning “signals” that are almost entirely independent of the training data. The first is the fact that the simple act of choosing a neural network architecture already injects a strong prior, which we call a deep (image) prior, on the solution to a learning problem. Because of this, problems such as inpainting and denoising can be solved effectively by a deep network even without access to any training data at all. Second, I will show that invariance to spatial transformations is sufficient to explain much of the performance of some self-supervised learning methods, even when these are applied to a single unlabelled training image, resulting in competitive pretraining of deep representations.

Single Image Generative Models: From Patch Based Methods to GANs

Tomer Michaeli & Tamar Rott Shaham, 15:25-15:55 UTC+1

https://youtu.be/tjxII31RegQ?t=7667

Despite the notable success of deep generative models (e.g. VAE, GANs), capturing the distribution of rich and diverse high resolution natural images is still considered a challenging task. In this talk we will show how the internal statistics of patches within a single natural image often carry enough information for learning a powerful generative model. We will start by showing this in the context of a rather unusual task: Detecting and visualizing tiny variations among repeating structures in a single image [dekel et. al, SIGGRAPH'15] or across multiple views [Tlusty et. al, CVPR'18]. Our methods allow attenuating those variations to obtain an ‘idealized’ version of the image, or magnifying them so as to produce an exaggerated image which highlights tiny differences between repeating structures. We will then show how these patch-based techniques can be naturally combined with powerful deep network frameworks, like GANs, allowing to learn a deep generative model from a single natural image [Rott Shaham et. al, ICCV'19]. We will demonstrate how this single-image GAN (SinGAN) framework can be used to solve many image manipulation and restoration tasks, and will survey exciting follow-up works exhibiting surprising uses of SinGAN.

Understanding and Extending Neural Radiance Fields

Jon Barron, 17:00-17:30 UTC+1

https://youtu.be/tjxII31RegQ?t=12174

Neural Radiance Fields (Mildenhall, Srinivasan, Tancik, et al., ECCV 2020) are an effective and simple technique for synthesizing photorealistic novel views of complex scenes by optimizing an underlying continuous volumetric radiance field, parameterized by a (non-convolutional) neural network. I will discuss and review NeRF and then introduce two works that closely relate to it: First, I will explain why NeRF (and other CPPN-like architectures that map from low-dimensional coordinates to intensities) depend critically on the use of a trigonometric "positional encoding", aided by insights provided by the neural tangent kernel literature. Second, I will show how NeRF can be extended to incorporate explicit reasoning about occluders and appearance variation, and can thereby enable photorealistic view synthesis and photometric manipulation using only unstructured image collections.

Texture is from Mars, Structure is from Venus: on combining internal and external learning

Alexei Efros, 17:30-18:00 UTC+1

https://youtu.be/tjxII31RegQ?t=14261