Video Segmentation via Object Flow

Yi-Hsuan Tsai Ming-Hsuan Yang Michael J. Black

University of California, Merced Max Planck Institute for Intelligent Systems

Overview of the proposed model. For segmentation, we consider a multi-level spatial-temporalmodel. Red circles denote pixels, which belong to the superpixel marked by the turquoise circles. The black and the red lines denote the spatial and temporal relationships, respectively. The relationships between the pixels and the superpixel are denoted by the turquoise lines. After obtaining the object mask, we use this mask to re-estimate the optical flow, and update both models iteratively.

Abstract

Video object segmentation is challenging due to fast moving objects, deforming shapes, and cluttered backgrounds. Optical flow can be used to propagate an object segmentation over time but, unfortunately, flow is often inaccurate, particularly around object boundaries. Such boundaries are precisely where we want our segmentation to be accurate. To obtain accurate segmentation across time, we propose an efficient algorithm that considers video segmentation and optical flow estimation simultaneously. For video segmentation, we formulate a principled, multiscale, spatio-temporal objective function that uses optical flow to propagate information between frames. For optical flow estimation, particularly at object boundaries, we compute the flow independently in the segmented regions and recompose the results. We call the process object flow and demonstrate the effectiveness of jointly optimizing optical flow and video segmentation using an iterative scheme. Experiments on the SegTrack v2 and Youtube-Objects datasets show that the proposed algorithm performs favorably against the other state-of-the-art methods.

Downloads

"Video Segmentation via Object Flow", Yi-Hsuan Tsai, Ming-Hsuan Yang, Michael J. Black, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016

[Paper] [Supplementary] [GitHub]

Results on the DAVIS benchmark dataset

Download: Our object masks and evaluation results

BibTex

@inproceedings{Tsai_CVPR_2016,

author = {Y.-H. Tsai and M.-H. Yang and M. J. Black},

booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},

title = {Video Segmentation via Object Flow},

year = {2016}}