Focus-based Metric Depth Estimation in Standard Plenoptic Cameras
In recent years, a lot of efforts have been devoted to the problem of depth estimation from lightfield images captured by plenoptic cameras. However, most of the depth estimation methods in the state-of-the-art do not provide metric information and depth estimates are delivered in method-specific units that are in general not linearly related to real-world dimensions. In this paper, we tackle the problem of focus-based metric depth estimation in standard plenoptic cameras. For this purpose we propose a closed-form model that relates the refocusing parameter with the focus distance of a plenoptic camera in order to allow for metric depth estimation. Based on the proposed model, we develop a calibration procedure that allows finding the parameters of the model. Using measurements of a time-of-flight sensor as ground-truth, experimental validation in a distance range of 0.2 to 1.6 m shows that focus-based depth estimation is feasible with a root-mean-squared error of less than 5 cm.
A standard plenoptic camera Lytro Illum can be used to capture a lightfield image of a scene. Based on the proposed calibration method, a metric depth map can be obtained using defocus information.
Download
Source code
Matlab code for lightfield decoding, refocusing, calibration and focus-based depth estimation. Instructions and sample images are included [download (93MB)]
Dataset 1
40 lightfield images of a checkerboard pattern. This images include depth ground-truth data captured with a TOF sensor in a distance range of 0.4 to 1.6m. These images are suitable for the calibration of standard plenoptic cameras and validation of metric focus-based depth estimation [download (2GB) ]
Dataset 2
11 lightfield images of a highly-textured planar test target with depth ground-truth captured with a TOF sensor in a distance range of 0.44 to 1.44 m approximately. Segmentation masks of the planar target are included. These images are suitable for the evaluation of metric depth estimation methods under controlled environments. [download (0.5GB) ]
Dataset 3
38 lightfield images of highly-textured scenes in a distance range of 0.2 to 1.6m. Depth ground-truth data captured with a TOF sensor is available. These images are suitable for the evaluation of depth estimation methods under controlled environments with depth discontinuities and multiple objects. [download (2GB) ]
Dataset 4
50 lightfield images of indoor scenes. Depth ground truth data captured with a TOF sensor and manual segmentation masks are available for each image. These images are suitable for the evaluation of depth estimation and segmentation algorthms for standard plenoptic cameras in challenging indoor scenarios. [download (2GB) ]
References
[1] S. Pertuz, E. Pulido-Herrera, J. K. Kamarainen, Focus Model for Metric Depth Estimation in Standard Plenoptic Cameras, ISPR Journal of Photogrammetry and Remote Sensing. 144:38-47, 2018. 10.1016/j.isprsjprs.2018.06.020