Event cameras are novel bio-inspired sensors, whose pixels work independently and asynchronously output intensity changes as events. Since the microsecond resolution (no motion blur) and high dynamic range (compatible with extreme light) of events, it is promising to directly segment objects from the sparse and asynchronous event stream in many applications. However, different from the rich cues in video object segmentation, it is challenging to segment complete objects from the sparse event stream. Given the object mask at the initial time, our task aims to segment complete object at any subsequent time in event streams. In this paper, we present the first recurrent framework for continuous-time object segmentation from event stream. To train and test ourt framework, we build a real-world event-based object segmentation dataset that contains event streams, APS images, and object annotations. Extensive experiments on our datasets demonstrate the effectiveness of the proposed recurrent architecture.
Our EOS dataset is recorded in a zoo thus the main objects are animals, it contains 167 sequences under different light intensities and motion. We provide the original event streams, the APS frames, the e2vid reconstruction frames and the object masks, the researchers can use our dataset to other segmentation tasks as they need (e.g., motion segmentation, unsupervised object segmentation). The object annotations are marked on the APS frames. The dataset is divided into training set and validate set, the training set is composed of 127 videos which have 16786 annotated frames, 1.7 objects and 3.68M events per sequences, and the Val-set comprises 40 videos consist 4541 annotated frames, 2.2 objects and 3.360M events per sequences.
Results on EOS Dataset
Results on Simulated Event-DAVIS Dataset
Results of different number of continuous events
Results of fixed number of continuous events with different shift window