DynaVol: Unsupervised Learning for Dynamic Scenes through Object-Centric Voxelization
Yanpeng Zhao*, Siyu Gao*, Yunbo Wang, Xiaokang Yang
MoE Key Lab of Artificial Intelligence, AI Institute
Shanghai Jiao Tong University, Shanghai 200240, China
Embedded Files
DynaVol_video.mp4Abstract
Unsupervised learning of object-centric representations in dynamic visual scenes is challenging. Unlike most previous approaches that learn to decompose 2D images, we present DynaVol, a 3D scene generative model that unifies geometric structures and object-centric learning in a differentiable volume rendering framework. The key idea is to perform object-centric voxelization to capture the 3D nature of the scene, which infers the probability distribution over objects at individual spatial locations. These voxel features evolve through a canonical-space deformation function, forming the basis for global representation learning via slot attention. The voxel and global features are complementary and leveraged by a compositional NeRF decoder for volume rendering. DynaVol remarkably outperforms existing approaches for unsupervised dynamic scene decomposition. Once trained, the explicitly meaningful voxel features enable additional capabilities that 2D scene decomposition methods cannot achieve: it is possible to freely edit the geometric shapes or manipulate the motion trajectories of the objects.
Page updated
Google Sites
Report abuse