Pixel-GCN for VOS

Motion-based Occlusion-aware Pixel Graph Network for

Video Object Segmentation

Saptakatha Adak (sapta@cse.iitm.ac.in) and Sukhendu Das (sdas@iitm.ac.in)

Visualization and Perception Lab, Dept. of Computer Science and Engineering

Indian Institute of Technology Madras, India

http://www.cse.iitm.ac.in/∼vplab

Abstract

This paper proposes a dual-channel based Graph Convolutional Network (GCN) for the Video Object Segmentation (VOS) task. The main contribution lies in formulating two pixel graphs based on the raw RGB and optical flow features. Both spatial and temporal features are learned independently, making the network robust to various challenging scenarios in real-world videos. Additionally, a motion orientation-based aggregator scheme efficiently captures long-range dependencies among objects. This not only deals with the complex issue of modelling velocity differences among multiple objects moving in various directions, but also adapts to change of appearance of objects due to pose and scale deformations. Also, an occlusion-aware attention mechanism has been employed to facilitate accurate segmentation under scenarios where multiple objects have temporal discontinuity in their appearance due to occlusion. Performance analysis on DAVIS-2016 and DAVIS-2017 datasets show the effectiveness of our proposed method in foreground segmentation of objects in videos over the existing state-of-the-art techniques. Control experiments using CamVid dataset show the generalising capability of the model for scene segmentation.

Paper

Motion-based Occlusion-aware Pixel Graph Network for Video Object Segmentation

In 26th International Conference on Neural Information Processing (ICONIP), Sydney, Australia, 2019.

[Paper] [Best Student Paper Award]

Architecture

Fig .1: Proposed Pixel-GCN framework for Video Object Segmentation.

Quantitative Results

i) Comparative result for Video Object Segmentation (VOS)

Table 1: Quantitative comparison of the proposed Pixel-GCN with existing state-ofthe-art methods on DAVIS-2016 and DAVIS-2017 validation datasets for VOS. “ ↑ ” - higher is better. “ ∗ ” - w/o proposed aggregation functions (Amotion & Aocc) in rows 9 & 10 (Best results in bold).

(ii) Comparative result for Video Semantic Segmentation (VSS)

Table 2: Quantitative comparison of various models on CamVid dataset having features as input for semantic segmentation in videos. “FS”: Feature → Segmentation, whereas “FF”: Feature → Feature. “ ∗ ” - without proposed aggregation functions (A_motion & A_occ) in rows 2 & 3
(Best results in bold). MO refers to the moving objects categories and SEG represents segmentation.