DeLiRa: Self-Supervised Depth,
Light, and Radiance Fields
Light, and Radiance Fields
Vitor Guizilini Igor Vasiljevic Jiading Fang Rares Ambrus
Sergey Zakharov Vincent Sitzmann Adrien Gaidon
Sergey Zakharov Vincent Sitzmann Adrien Gaidon
Abstract. Differentiable volumetric rendering is a powerful paradigm for 3D reconstruction and novel view synthesis. However, standard volume rendering approaches struggle with degenerate geometries in the case of limited viewpoint diversity, a common scenario in robotics applications. In this work, we propose to use the multi-view photometric objective from the self-supervised depth estimation literature as a geometric regularizer for volumetric rendering, significantly improving novel view synthesis without requiring additional information. Building upon this insight, we explore the explicit modeling of scene geometry using a generalist Transformer, jointly learning a radiance field as well as depth and light fields with a set of shared latent codes. We demonstrate that sharing geometric information across tasks is mutually beneficial, leading to improvements over single-task learning without an increase in network complexity. Our DeLiRa architecture achieves state-of-the-art results on the ScanNet benchmark, enabling high quality volumetric rendering as well as real-time novel view and depth synthesis in the limited viewpoint diversity setting.
Contributions:
We show that the multi-view photometric objective is an effective regularization tool for volumetric rendering, as a way to mitigate the shape-radiance ambiguity.
We propose a novel architecture for the joint learning of depth, light, and radiance fields, decoded from a set of shared latent codes.
Our proposed method achieves state-of-the-art view synthesis and depth estimation results on the ScanNet benchmark, outperforming methods that require explicit supervision or pretrained networks.
Citation
@inproceedings{tri_delira,
author = {Vitor Guizilini and Igor Vasiljevic and Jiading Fang and Rares Ambrus and Sergey Zakharov and Vincent Sitzmann and Adrien Gaidon},
title = {DeLiRa: Self-Supervised Depth, Light, and Radiance Fields},
booktitle = {International Conference on Computer Vision (ICCV)},
year = {2023},
}