NeuPhysics: Editable Neural Geometry and Physics from Monocular Videos
University of Maryland, College Park
* Equal Contribution
36th Conference on Neural Information Processing Systems (NeurIPS 2022)
Abstract
We present a method for learning 3D geometry and physics parameters of a dynamic scene from only a monocular RGB video input. To decouple the learning of underlying scene geometry from dynamic motion, we represent the scene as a time-invariant signed distance function (SDF) which serves as a reference frame, along with a time-conditioned deformation field. We further bridge this neural geometry representation with a differentiable physics simulator by designing a two-way conversion between the neural field and its corresponding hexahedral mesh, enabling us to estimate physics parameters from the source video by minimizing a cycle consistency loss. Our method also allows a user to interactively edit 3D objects from the source video by modifying the recovered hexahedral mesh, and propagating the operation back to the neural field representation. Experiments show that our method achieves superior mesh and video reconstruction of dynamic scenes compared to other competitive Neural Field approaches, and we provide extensive examples which demonstrate its ability to extract useful 3D representations from videos captured with consumer-grade cameras.
Overview
NeuPhysics takes a monocular video as supervision. The rigidity, color, and SDF are defined on a canonical frame, while the motion of each spatial point is time-dependent. After fusing the two streams, volume rendering is used to reconstruct images and compute the color loss. A differentiable simulator is embedded after the neural fields to learn the dynamics parameters and edit the scene.
3D Reconstruction from Monocular Videos
Source Video
Ours
D-NeRF [1]
NeuS [2]
Source Video
Ours
D-NeRF [1]
NeuS [2]
Source Video
Ours
D-NeRF [1]
NeuS [2]
Video Editing
Delete Foreground
Highlight Foreground
Delete Foreground
Highlight Foreground
Estimated Material
Hard Material
Soft Material
Estimated Velocity and Gravity
Add Rightward Velocity
Add Leftward Velocity
Reference
[1] Pumarola, Albert, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-NeRF: Neural radiance fields for dynamic scenes. CVPR 2021.
[2] Wang, Peng, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. "NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction." NeurIPS 2021.
Bibtex
@inproceedings{qiao2022neuphysics,
author = {Qiao, Yi-Ling and Gao, Alexander and Lin, Ming C.},
title = {NeuPhysics: Editable Neural Geometry and Physics from Monocular Videos},
booktitle = {Conference on Neural Information Processing Systems (NeurIPS)},
year = {2022},
}