Depth image enhancement using perceptual texture priors

People

Duhyeon Bang and Hyunjung Shim

Abstract

A depth camera, also is called range image camera, is widely used in various applications because it provides a distance information of the scene in real time. However, due to the limitation of power consumption or hardware, the depth camera presents severe noises, incapable of providing the high quality 3D data.

Although the smoothness prior is often employed to subside the depth noise, it discards the geometric details so to degrade the distance resolution. Increasing contrast is also used for enhancing perceptual quality, but it causes distortion of details in the scene. In other words, these existing techniques hinder achieving the realism in 3D contents.

Since humans are usually aware whether the scene is real from detecting details in the scene, we focus on the restoration of the geometric details, especially textures, to make observers feel realism. We propose a perceptual-based depth image enhancement technique that automatically recovers the depth details of various textures, using a statistical framework inspired by human mechanism of perceiving surface details by texture priors.

Our concept is replacing bad normals data taken from the depth camera with fine normals data that is best matched one with input from the database. To do this, First we build the database. Since we focus on enhancing details of object's texture, the database is composed of the texture image whit it's high quality normals data.

Next, we make an algorithm to find the best matched one with input automatically. Since the color image has more information of the texture than normals data, we use the input color image for searching the best matched one from the database. We use principle components analysis(PCA) and select the pattern density as a primary feature to classify textures, Based on the recent studies in human visual perception (HVP).

Finally, If we find the best matched one from the database, we replace the normals by using existing method.

Overview

We use the cushion and the backrest image in the office scene. These objects are marked by a red and yellow box respectively. Since the pattern density affects the perceptual roughness, we decide to use the pattern density as our primary feature for classifying the textures. First, we train the classifiers to categorize various surfaces based on HVP. After that, we segment the input color image into several local regions upon corresponding depth image, apply classifiers to select target regions and identify their best match from the database. Finally, we replace the input noisy normals by the selected high resolution normals. We use the existing method( D Nehab et al. “Efficiently combining positions and normal for precise 3D geometry” .Proc. ACM SIGGRAPH; 536-543 (2005). http://w3.impa.br/~diego/software/NehEtAl05/).

Results

We obtain a pair of color and depth map from a commercial depth sensor, Kinect 2.0.

Results of geometric enhancement. (a), (e): The input of a cushion marked with the red box and the input of a backrest marked with the yellow box in Fig. 2. (b), (f): The selected images from database. (c), (g): The computed normals form input depth map. (d), (h): The enhanced depth data are in the right bottom respectively.

- Input data:

◦Some missing holes and distortions are observed in Fig. 3(c) and (g).

- Improvement

◦Fig. 3(d): Not only enhancing the depth precision and geometric details, but also restoring missing holes.

◦Fig. 3(g): correctly retaining the original structure of object surface.

Publication

Depth image enhancement using perceptual texture priors

D Bang, H Shim, SPIE Human Vision and Electronic Imaging, 2015, Feb.