Di-NeRF

Di-NeRF: Distributed NeRF for Multi-Robot Collaborative Learning with Relative Poses Refinement

Mahboubeh Asadi*, Kourosh Zareinia*, Sajad Saeedi*

* Toronto Metropolitan University

IEEE-RA-L 2024 - ICRA 2025

Di-NeRF: Distributed NeRF for Collaborative Learning with Unknown Relative Poses

Code (coming soon)

Supplementary_document_Di_NeRF.pdf

Supplementary Document - Di-NeRF

ieeexplore.ieee.org/document/10705057

Di-NeRF

Abstract

Collaborative mapping of unknown environments can be done faster and more robustly than a single robot. However, a collaborative approach requires a distributed paradigm to be scalable and deal with communication issues. This work presents a fully distributed algorithm enabling a group of robots to collectively optimize the parameters of a Neural Radiance Field (NeRF). The algorithm involves the communication of each robot's trained NeRF parameters over a mesh network, where each robot trains its NeRF and has access to its own visual data only. Additionally, the relative poses of all robots are jointly optimized alongside the model parameters, enabling mapping with less accurate relative camera poses. We show that multi-robot systems can benefit from differentiable and robust 3D reconstruction optimized from multiple NeRFs. Experiments on real-world and synthetic data demonstrate the efficiency of the proposed algorithm.

Di-NeRF is open-source (code coming soon)

ICRA25_4777.mp4

Di-NeRF-Video-2.mov

Experiment

In the work, three datasets are used to evaluate Di-NeRF: The synthetic dataset, Tanks and Temples, and the Waymo dataset.

Di-NeRF vs Centralized NeRF on Synthetic Dataset

For the synthetic sequences, two robots are involved, and the dataset is split into two segments. For instance, in the case of the Chair sequence, robot R#1 exclusively observes the front of the chair. In contrast, robot R#2 focuses solely on the back (without any overlap in the robots' trajectories). A similar pattern is applied to other synthetic sequences. The whole data are divided into two parts for synthetic dataset and with COLMAP the poses were estimated for each segment. Therefore, the coordinate system for each robot is different, and the relative pose can be estimated by running Di-NeRF.

Raw data provided for robot#1

Raw data provided for robot#2

Independent optimization for robot#1

Independent optimization for robot#2

In the following cases, the local origins for robot R#1 and robot#2 are different.

Di-NeRF without pose optimization for robot#1

Di-NeRF without pose optimization for robot#2

Di-NeRF for robot#1

Di-NeRF for robot#2

Relative pose optimization for two robots, red poses are for robot#1 and blue poses are for robot#2 which is optimized to transform to robot#1 local origin. There are 20 percent overlap in the trajectory.

Di-NeRF allows robots to cooperatively optimize local copies of a neural network model without explicitly sharing visual data. In this figure, two robots use Di-NeRF to cooperatively optimize a unified NeRF. Each robot only sees part of the chair, and robots do not know their relative poses. The robots communicate over a wireless network (gray dashed lines) to cooperatively optimize the final network and relative poses.

Di-NeRF can reconstruct the entire scene via the collaboration of two robots, where each robot only sees a part of the scene. Individual training results in poor reconstruction quality for some areas of the scene, whereas Di-NeRF maintains good quality throughout. In each column, the left image is for robot R#1 and the right one is for robot R#2.

Tanks and Temples - Barn Dataset

(a) sample images, from the Barn sequence, provided to each robot; (b, c, d) sample images generated by Di-NeRF, once the robots collaboratively learn the scene. The images are strategically chosen—each is only visible in the raw data of a specific robot but can be rendered by all robots.

Di-NeRF for different numbers of robots (all fully connected); (left) the allocation of frames and poses to the robots; (right) none of the robots have common images, but all robots can render the whole scene.

The San Francisco Mission Bay Waymo Dataset - Unbounded Scenes

For this experiment, a segment of the data, including 233 images from one camera (one of 8 that provide a complete surround view from the roof of the car), measuring approximately 286 meters, was selected. The rendering result for this setup is shown for centralized and Di-NeRF training. The average values of PSNR and SSIM are 25.60 and 0.831 over 6 robots and 25.39 and 0.848 for the baseline.