PRL2022LFCoding

A Hybrid Tucker-VQ Tensor Sketch Decomposition Model for Coding and Streaming Real World Light Fields using Stack of Differently Focused Images

Joshitha R, Mansi Sharma, Sally Khaidem

Pattern Recognition Letters, 2022

Abstract

Computational multi-view displays involving light fields are a fast emerging choice for 3D presentation of real-world scenes. Tensor autostereoscopic glasses-free displays use just few light attenuating layers in front of a backlight to output high quality light field. We propose three novel schemes, Focal Stack - Hybrid Tucker-TensorSketch Vector Quantization (FS-HTTSVQ), Focal Stack - Tucker-TensorSketch (FS-TTS), and Focal Stack - Tucker Alternating Least-Squares (FS-TALS), for efficient representation, streaming and coding of light fields using a stack of differently focused images. Working with a focal stack instead of the entire light field majorly reduces the data acquisition cost as well as the computation and processing cost. Extensive experiments with real world light field focal stacks demonstrate that proposed novel one-pass Tucker decomposition using TensorSketch with hybrid vector quantization in FS-HTTSVQ, compactly represents the approximated focal stack in codebook form for better transmission and streaming. Encoding with High Efficiency Video Coding (HEVC) eliminates all intrinsic redundancies present in the approximated focal stack. Resultant low-rank approximated and coded focal stack is then employed to analytically optimize layer patterns for the tensor display. The complete end-to-end light field processing pipelines flexibly work for multiple bitrates and are adaptable for a variety of multi-view autostereoscopic platforms. Our schemes exhibit note-worthy performances on focal stacks compared to direct encoding of an entire light field using a standard codec like HEVC.

PRL2022LF.pdf

Citation

Joshitha Ravishankar, Mansi Sharma, Sally Khaidem, "A Hybrid Tucker-VQ Tensor Sketch decomposition model for coding and streaming real world light fields using stack of differently focused images," Pattern Recognition Letters, Volume 159, 2022, Pages 23-30, ISSN 0167-8655, https://doi.org/10.1016/j.patrec.2022.04.034.

BibTex

@article{RAVISHANKAR202223,

title = {A Hybrid Tucker-VQ Tensor Sketch decomposition model for coding and streaming real world light fields using stack of differently focused images},

journal = {Pattern Recognition Letters},

volume = {159},

pages = {23-30},

year = {2022},

issn = {0167-8655},

doi = {https://doi.org/10.1016/j.patrec.2022.04.034},

url = {https://www.sciencedirect.com/science/article/pii/S0167865522001465},

author = {Joshitha Ravishankar and Mansi Sharma and Sally Khaidem},

keywords = {Light field, Focal stack, Low-rank representation, Tucker decomposition, Tensor sketching, Vector quantization, Compression, Streaming, Tensor display, HEVC}