Video Retargeting

Spatial-Temporal Saliency Map-based Video Retargeting by Dynamic Programming

Video retargeting is a process of transforming an existing video to fit the dimensions of an arbitrary display for a better quality of experience (QoE) in video adaptation. We propose and implement a saliency map-based cropping-and-scaling system. The saliency map is the metric to measure the importance of pixels in each frame which consists of both spatial and temporal (motion) attention features. The spatial attention is computed in frequency domain of a color image by quaternion Fourier transforms (QFT) on phase spectrum. The motion attention is calculated by the residual of global motion compensation, where the global motion is robustly estimated with either a direct (pixel-based) method or a feature (such as KLT-tracking features or SIFT features) -based method. Both attention models are fused by a nonlinear method. The cropping-and-scaling retargeting is more efficient due to low computation cost and good trade-off on the eventual performance. The retargeting framework is constructed based on the spatial-temporal optimization process. In the optimization, the sum of cropping and scaling information loss is minimized where the scaling loss is measured with a result of down-sampling-and-up-sampling. To keep temporal coherence, the dynamic programming is applied for each shot in the video, which realizes the global optimization in cropping location and scale estimation.

Demo