Robust Approach to Inverse Lighting using RGB-D images

People

Junsuk Choe and Hyunjung Shim

Abstract

Proper lighting is a key element in developing a photorealistic computer-generated image. This paper introduces a novel approach for robustly extracting lighting conditions from an RGB-D (RGB + depth) image. Existing studies on lighting estimation have developed image analysis techniques by constraining the scope and condition of the target objects. For example, they have assumed that the objects have homogeneous surfaces, inter-reflections can be ignored, and their three-dimensional (3D) geometries consist of a noise-free mesh. These assumptions, however, are unrealistic; real objects often have complicated non-homogeneous surfaces, inter-reflections that affect a considerable portion of illumination, or unpredictable noise that can affect sensor measurements. To overcome these limitations, this study takes non-homogeneous surface objects into account in the inverse lighting framework via segment-based scene representation. Moreover, we employ outlier removal and appropriate region selection to achieve robust lighting estimation in the presence of inter-reflections and noise. We demonstrate the effectiveness of the proposed approach by conducting extensive experiments on synthetic and real RGB-D images.

Fig 1. Block diagram of proposed approaches

Results

Fig 2. Experimental results of inverse lighting

Fig 3. Experimental results of inserting objects using estimated lighting

URLs

Paper [PDF] [Elsevier]

Related Work

Inverse Lighting for Non-homogeneous Objects from Color and Depth Image using Wavelet Represnetation

Junsuk Choe and Hyunjung Shim.

IS&T/SPIE Electronic Imaging, 2015.

(Poster Presentation) [Poster] [PDF]

Acknowledgements

We thank three reviewers for their insightful comments and constructive suggestions, which greatly improve our paper. This work was supported by ICT R&D program of MSIP/IITP [ R7124-16-0004, Development of Intelligent Interaction Technology Based on Context Awareness and Human Intention Understanding ], the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the MSIP (NRF-2016R1A2B4016236), the MSIP (Ministry of Science, ICT and Future Planning), Korea, under the University Information Technology Research Center support program (IITP-2017-2016-0-00288) supervised by the IITP (Institute for Information & communication Technology Promotion), and the MSIP (Ministry of Science, ICT and Future Planning), Korea, under the IT Consilience Creative Program (IITP-2017-2017-0-01015) supervised by the IITP(Institute for Information & Communications Technology Promotion).