ParticleNeRF: A Particle-Based Encoding for Online Neural Radiance Fields
WACV24 Best Paper (Honorable Mention)
Jad Abou-Chakra, Feras Dayoub, Niko Suenderhauf
Queensland University of Technology, University of Adelaide
Abstract
While existing Neural Radiance Fields (NeRFs) for dynamic scenes are offline methods with an emphasis on visual fidelity, our paper addresses the online use case that prioritises real-time adaptability. We present ParticleNeRF, a new approach that dynamically adapts to changes in the scene geometry by learning an up-to-date representation online, every 200ms. ParticleNeRF achieves this using a novel particle-based parametric encoding. We couple features to particles in space and backpropagate the photometric reconstruction loss into the particles' position gradients, which are then interpreted as velocity vectors. Governed by a lightweight physics system to handle collisions, this lets the features move freely with the changing scene geometry. We demonstrate ParticleNeRF on various dynamic scenes containing translating, rotating, articulated, and deformable objects. ParticleNeRF is the first online dynamic NeRF and achieves fast adaptability with better visual fidelity than brute-force online InstantNGP and other baseline approaches on dynamic scenes with online constraints.Â
Video
Method
A query point is sampled in space. The features and positions of the particles within a search radius are retrieved. The features and distances from the query point are used to interpolate the feature at the query point. The resulting feature is evaluated by the neural network to give color and density. To train the encoding, the loss gradients are backpropagated through the network, the query feature, and finally into the positions and features of the particles.
Particles
The above GIF shows all the particles contributing the to the geometry of the object.
When we move the object, the particles follow. This shows that the NeRF loss propagating back into the particle positions is effective at tracking the object. Note that we have to retrigger the rendering after the motion which is why the GIF shows a delay in the update.
Example of Particle Pruning and Growing Strategy
Starting with small number of particles
Starting with high number of particles
Dynamic Dataset
Animated Blender Dataset
