DeLiRa: Self-Supervised Depth,
Light, and Radiance Fields

Vitor Guizilini     Igor Vasiljevic     Jiading Fang     Rares Ambrus   
Sergey Zakharov     Vincent Sitzmann     Adrien Gaidon

Abstract. Differentiable volumetric rendering is a powerful paradigm for 3D reconstruction and novel view synthesis. However, standard volume rendering approaches struggle with degenerate geometries in the case of limited viewpoint diversity, a common scenario in robotics applications. In this work, we propose to use the multi-view photometric objective from the self-supervised depth estimation literature as a geometric regularizer for volumetric rendering, significantly improving novel view synthesis without requiring additional information. Building upon this insight, we explore the explicit modeling of scene geometry using a generalist Transformer, jointly learning a radiance field as well as depth and light fields with a set of shared latent codes. We demonstrate that sharing geometric information across tasks is mutually beneficial, leading to improvements over single-task learning without an increase in network complexity. Our DeLiRa architecture achieves state-of-the-art results on the ScanNet benchmark, enabling high quality volumetric rendering as well as real-time novel view and depth synthesis in the limited viewpoint diversity setting.

Contributions:

• We show that the multi-view photometric objective is an effective regularization tool for volumetric rendering, as a way to mitigate the shape-radiance ambiguity. 

• We propose a novel architecture for the joint learning of depth, light, and radiance fields, decoded from a set of shared latent codes. 

• Our proposed method achieves state-of-the-art view synthesis and depth estimation results on the ScanNet benchmark, outperforming methods that require explicit supervision or pretrained networks.

Citation

@inproceedings{tri_delira,

  author = {Vitor Guizilini and Igor Vasiljevic and Jiading Fang and Rares Ambrus and Sergey Zakharov and Vincent Sitzmann and  Adrien Gaidon},

  title = {DeLiRa: Self-Supervised Depth, Light, and Radiance Fields},

  booktitle = {International Conference on Computer Vision (ICCV)},

  year = {2023},

}