View Synthesis

View Synthesis/Interpolation with One or Two Views plus Respective Depth Maps

This page contains my past work of view synthesis from one or two views plus respective depth maps in the project "3-D Videos of Multiple Views", Thomson Corporate Research, Princeton.

Abstract:

View Synthesis is an exciting technique in image based rendering (IBR). It includes some critical steps as pre-processing of depth map, 3-d image warping and hole filling etc. In 3-d image warping, the transform consists of reprojection (2D-to-3D) and subsequent backprojection (3D-to-2D). In new view generation, two challenging problems should be handled, i.e. visibility (folds) and de-occlusions (holes) . Here the visibility is verified by Z-buffer rather than a scanning order arrangement, and "holes" are filled by a layered-based rendering scheme, where each reference view is represented by a main layer, a boundary layer and a background layer. (Note: The testing data is multiple view videos "Breakdancer" and "Ballet" from Microsoft Research Lab, with known calibration parameters and depth maps.)

Splatting or Meshing for Rendering/Reconstruction

Splat is the pixel shape projected to reconstruct the new view in the image space where Footprint is the reconstruction kernel. A depth map represents a cloud of 3d points to be splatted, and splatting broaden the individual 3d points.

Splatting for 3-D image rendering

In meshing, pixel plus depth is regarded as vertex of a triangular mesh and rendering is done by interpolation within meshes. Depth discontinuity (phantom/rubber sheet) has to be handled by breaking all edges with large difference and removing those corresponding triangles.

Triangular mesh-based 3-d image rendering