Multi-view synthesis

Typical 3D video system with “video-plus-depth” data

• At the sender side, depth data is estimated with a limited number of input views (e.g., 1-3 views) or captured with depth camera.

• At the receiver side, the video and depth data are decoded and the view synthesis is used to generate as many additional views as required by the display (e.g., 9, 16, 28 views).

View synthesis techniques are also very important:

• For free viewpoint display, the view synthesis is used to generate as many additional views as required by the display.

• Perception based view synthesis technique is required to provide viewers with good depth quality and visual comfort

Three key components for multi-view video service: 

• (1) Depth estimation, (2) 3D video coding, (3) View synthesis

“Depth Image-Based Rendering (DIBR)”

• Depth map processing for depth estimation error reduction

  • Discontinuity preserving smoothing using joint filtering

  • Object boundary detection & dilation, Depth image hole filling

• 3D warping & image hole filling

  • Disocclusions appear in the virtual view.

    • Wide angle view -> large disocclusion regions (difficult to be filled)

  • Multi-view video inpainting is required.