Y. Benezeth, D. Sidibé, J.-B. Thomas, "Background subtraction with multispectral video sequences",  12th OMNIVIS workshop of the International Conference on Robotics and Automation (ICRA), 2014.  (pdf)

Overview

Motion analysis of moving targets is an important issue in several applications such as video surveillance. Background subtraction is one of the simplest and widely used techniques for moving target detection in video sequences. In this project, we investigate the advantages of using a multispectral video acquisition system of more than three bands for background subtraction over the use of trichromatic video sequences. To this end, we have established a dataset of multispectral videos with a manual annotation of moving objects. To the best of our knowledge, this is the first publicly available dataset of multispectral video sequences.

More information about background subtraction methods, their possible extension to multispectral videos and a quantitative comparison of background subtraction algorithms on regular color videos and on multispectral videos is presented in the OMNIVIS'2014 paper .

Presentation of the dataset

The videos

Our dataset is composed of 5 video sequences containing between 250 and 2300 frames of size 658 в 491. The video dataset represents 1 indoor video sequence and 4 outdoor scenes with different challenges such as gradual illumination changes, shadows, camouflage effects (color similarity of object and background) and intermittent object motion.

For each scene, we have acquired a sequence of multispectral images whose frame rate depends on the overall scene illumination, i.e. from 5 frames per second for dark scenes to 15 frames per second for bright ones. The acquisitions are performed with a commercial camera from FluxData, Inc. (the FD-1665-MS). This camera can acquire 7 spectral narrow bands, six in the visible spectrum and one in the near infrared. Color image sequences are easily obtained with a linear integration of the original multispectral images weighted by three different spectral envelopes.

The annotations

In order to propose fair quantitative results, it is required to annotate the video dataset at pixel resolution. However, because such a precise annotation is very time consuming, most datasets in the literature annotate moving objects with bounding boxes. In this project, we propose the first dataset that is composed of several registered multispectral and color image sequences. These videos have been manually annotated at pixel resolution using a software provided by the authors of the changedetection workshop. We use the following labels:

• static pixels are assigned grayscale value of 0,

• moving pixels are assigned grayscale value of 255,

• pixels outside the Region Of Interest (ROI) are assigned grayscale value of 85,

• unknown pixels are assigned grayscale value of 170.

In total, more than 7400 frames have been manually annotated with these labels. Two examples of annotation with cars and pedestrian are presented below.

The format of the videos

Each video is composed of registered multispectral and color image sequences. The multispectral frames are provided as multi-page TIFF images with 7 bands. These multispectral images can be easily read with Matlab using:

   for k = 1:7

     myMS(:,:,k) = imread('in000001.tif', k);

   end

Download

The 5 annotated video sequences used in the paper can be downloaded here (about 21Gb). Each video folder is composed of the multispectral video, the trichromatic video and the annotations. 

Please note that we also share additional videos without annotations...

Contact

Yannick Benezeth (Université de Bourgogne), yannick.benezeth@u-bourgogne.fr

Désiré Sidibé (Université de Bourgogne), dro-desire.sidibe@u-bourgogne.fr

Jean-Baptiste Thomas (Université de Bourgogne), jean-baptiste.thomas@u-bourgogne.fr