Abstract. Differentiable volumetric rendering is a powerful paradigm for 3D reconstruction and novel view synthesis. However, standard volume rendering approaches struggle with degenerate geometries in the case of limited viewpoint diversity, a common scenario in robotics applications. In this work, we propose to use the multi-view photometric objective from the self-supervised depth estimation literature as a geometric regularizer for volumetric rendering, significantly improving novel view synthesis without requiring additional information. Building upon this insight, we explore the explicit modeling of scene geometry using a generalist Transformer, jointly learning a radiance field as well as depth and light fields with a set of shared latent codes. We demonstrate that sharing geometric information across tasks is mutually beneficial, leading to improvements over single-task learning without an increase in network complexity. Our DeLiRa architecture achieves state-of-the-art results on the ScanNet benchmark, enabling high quality volumetric rendering as well as real-time novel view and depth synthesis in the limited viewpoint diversity setting.
Contributions:
We show that the multi-view photometric objective is an effective regularization tool for volumetric rendering, as a way to mitigate the shape-radiance ambiguity.
We propose a novel architecture for the joint learning of depth, light, and radiance fields, decoded from a set of shared latent codes.
Our proposed method achieves state-of-the-art view synthesis and depth estimation results on the ScanNet benchmark, outperforming methods that require explicit supervision or pretrained networks.
Citation
@inproceedings{tri_delira,
author = {Vitor Guizilini and Igor Vasiljevic and Jiading Fang and Rares Ambrus and Sergey Zakharov and Vincent Sitzmann and Adrien Gaidon},
title = {DeLiRa: Self-Supervised Depth, Light, and Radiance Fields},
booktitle = {International Conference on Computer Vision (ICCV)},
year = {2023},
}
