MERLiN: Single-Shot Material Estimation and Relighting for Photometric Stereo 

 Abstract

Photometric stereo typically demands intricate data acquisition setups involving multiple light sources to recover surface normals accurately. In this paper, we propose MERLiN, an attention-based hourglass network that integrates single image-based inverse rendering and relighting within a single unified framework. We evaluate the performance of photometric stereo methods using these relit images and demonstrate how they can circumvent the underlying challenge of complex data acquisition. Our physically-based model is trained on a large synthetic dataset containing complex shapes with spatially varying BRDF and is designed to handle indirect illumination effects to improve material reconstruction and relighting. Through extensive qualitative and quantitative evaluation, we demonstrate that the proposed framework generalizes well to real-world images, achieving high-quality shape, material estimation, and relighting. We assess these synthetically relit images over photometric stereo benchmark methods for their physical correctness and resulting normal estimation accuracy, paving the way towards single-shot photometric stereo through physically-based relighting. This work allows us to address the single image-based inverse rendering problem holistically, applying well to both synthetic and real data and taking a step towards mitigating the challenge of data acquisition in photometric stereo.

Acknowledgements

Special thanks to the anonymous reviewers for their constructive feedback on our work. We would like to acknowledge the generous support of the Prime Minister Research Fellowship (PMRF) grant, the Japan Society for the Promotion of Science (JSPS) KAKENHI Grant (Number JP24K02966), and the Jibaben Patel Chair in Artificial Intelligence for the completion of this work.