Spherical Video

under construction...

360 Degree Spherical Video Generation, Rendering and Navigation

1. Image Mosaic

•The mosaic has a natural interpretation in 3D

•the images are reprojected onto a common plane

•the mosaic is formed on this plane

•mosaic is a synthetic wide-angle camera

2. Perspective Panorama

•Pick one image, typically the central view (red outline);

•Warp the others to its plane:

•Pixel-based or feature-based registration;

•Global alignment: bundle adjustment.

•Blend (global and local): alpha blending/Poisson blending/multi-band blending.

3. What if a 360 Field of View?

•Mapping to non-planar (e.g., cylindrical or spherical) surfaces;

•Lens distortion: radial or tangential;

•Fisheye lens: a different model than polynomial models (radial);

•Local registration for de-ghosting:

•optic flow or spline-based,

•Model of planar plus parallax;

4. Cylindrical Panorama

•Map 3D point (X,Y,Z) onto cylinder;

•Convert to coordinates on unit cylinder;

•Convert to image coordinates on unwrapped cylinder.

•A cylindrical image is a rectangular array.

5. Spherical Panorama

•Map 3D point (X,Y,Z) onto sphere;

•Convert to spherical coordinates;

•Convert to spherical image coordinates;

•Map onto the unwrapped sphere.

6. Image Blending

•Directly averaging the overlapped pixels results in ghosting artifacts

•Moving objects, errors in registration, parallax, etc.

•Assign one input image each output pixel

•One solution: optimal assignment can be found by graph cut

•Inconsistency btw pixels from different input images

•Different exposure/white balance settings

•Photometric distortions (e.g., vignetting)

•Poisson blending

•Copy the gradient field from the input image;

•Reconstruct the final image by solving a Poisson equation;

•Laplacian pyramid blending;

•Two-band and multi-band blending;

•Color correction and HDR for seam finding.

7. Examples of Planar, Cylindrical and Spherical Panorama

Planar

Spherical

Original images

Planar

Cylindrical

Spherical

Cylindrical

Spherical

Cylindrical

Spherical

8. 360 Spherical View Navigation

•Saved as equi-rectangular (or spherical) image: a 360x180 flat projection of a sphere;

•Html5/WebGL support viewing a spherical image by vertex + fragment shaders;

•360 viewer/player can map the equi-rectangular image as texture with a spherical surface mesh to image plane;

•YouTube supports 360 spherical video playback with 2 DoF (no zooming).

Equirectangular 360 degree image (Red: field of view) Image view with given fov (zoom) and pan, tilt angles

Appendix: Stereo Spherical Panorama Player

Stereo Format: Left and Right

Stereo Format: Up and Down

Left Right

Note: The rotations (panning and tilting) are not compensated yet.