Radiance fields have gradually become a main representation of media. Although its appearance editing has been studied, how to achieve view-consistent recoloring in an efficient manner is still under explored. We present RecolorNeRF, a novel user-friendly color editing approach for the neural radiance fields. Our key idea is to decompose the scene into a set of pure-colored layers, forming a palette. By this means, color manipulation can be conducted by altering the color components of the palette directly. To support efficient palette-based editing, the color of each layer needs to be as representative as possible. In the end, the problem is formulated as an optimization problem, where the layers and their blending weights are jointly optimized with the NeRF itself. Extensive experiments show that our jointly-optimized layer decomposition can be used against multiple backbones and produce photo-realistic recolored novel-view renderings. We demonstrate that RecolorNeRF outperforms baseline methods both quantitatively and qualitatively for color editing even in complex real-world scenes.

In this paper, we propose a novel method, RecolorNeRF, to conduct photo-realistic palette-based color editing on NeRF representations. We use “over” composition as the imaging formulation. Specifically, the appearance of each point in a 3D scene is represented by an alpha blending of a set of ordered pure-colored layers, which form a palette for editing. To do this, we model the layer opacity as a volumetric alpha field for each layer. Our proposed approach also includes the first trial of geometric palette optimization, which regularizes the palette to convexly span the color space of the 3D scene. Furthermore, in order to encourage the independence and representativeness of the learnable palette, sparsity of the blending weights is imposed through our novel soft sparsity norm and order-dependent weighting scheme. The layer decomposition, palette, and the volumetric radiance fields are optimized in a unified framework. The figure above shows the network pipeline of our method.

Our RecolorNeRF can robustly decompose 3D scenes into multiple layers and enable diverse high-fidelity recoloring of complex 3D scenes in an efficient manner, without any restriction to backbone NeRF models, fine-tuning, and additional deep feature extraction.