Tracker Benchmark v1.0

The visual tracker benchmark is hosted at http://www.visual-tracking.net .
Python version of the benchmark code is available at https://github.com/jwlim/tracker_benchmark.

[ Outdated ] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

This benchmark includes the results from 50 test sequences and 29 trackers. The results are reported in the CVPR 2013 paper [Paper][Supplement].

@inproceedings{ WuLimYang13,
  Title = {Online Object Tracking: A Benchmark},
  Author = {Yi Wu and Jongwoo Lim and Ming-Hsuan Yang},
  Booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  Year = {2013}
}

How to get the tracker benchmark codebase.
The tracker codes used in this benchmark can be download: tracker_benchmark_v1.0.zip (229MB).
If you suffer from slow download speed, try this link to the file on Google Drive.
The benchmark results using the above code is available also : tracker_benchmark_v1.0_results.zip (222MB, [Google Drive]).
The results zip-file needs to be unzipped *in* the ‘tracker_behcnmark_v1.0′ directory.
Join visual-tracking Google groups for further updates and discussions.

How to get the test data.
Simply, you can download the test sequence from the [zip] link in the test sequences table below.
For evaluating your tracker, you can find the ground-truth markings as well as MATLAB functions for evaluation in the the benchmark codebase.

How to compare your result with the benchmark results.
The benchmark results are reported as success plots and precision plots.
Note that the reported performance is the average over all sequences in the category. The test sequences table and attributes table shows which sequence has which attributes. For the detailed information about evaluation, refer the CVPR 2013 paper above.

Consider the performance of each tracker reported in this site as a reasonable lower-bound performance.
We used one default parameter (used in the original implementation) for all sequences without any tuning.
In later versions the performance of the trackers may be updated.
However, we still do NOT allow hand-tuning and using different parameters for individual sequences.

The protocol for tracker evaluation using the benchmark data:

  • Tracking starts from one initial bounding box in a start frame.
    The start frame may not be the first frame of the sequence.
  • DO NOT use manually-tuned, different parameters for individual sequences.
    Any additional information, such as sequence name or attributes, may not be used in determining the parameters.
    It is allowed for a tracker to automatically adapt its parameters only using features from input images.
  • The tracking result is a sequence of bounding boxes at each frame.
    Affine or similarity parameters are converted into bounding boxes for comparison.
  • Use the AUC (area under curve) to show the tracker’s overall performance.
  • The success plots are preferred over the precision plots, since precision only uses the bounding box locations, and ignores the size or overlap.

Test sequences

The benchmark contains 50 sequences from recent literatures.
The sequence names are in CamelCase without any blanks or underscores (_).
When there exist multiple targets each target is identified as dot+id_number (e.g. Jogging.1 and Jogging.2).
Each row in the ground-truth files represents the bounding box of the target in that frame, (x, y, box-width, box-height).
In most sequences the first row corresponds to the first frame and the last row to the last frame, except the following sequences:
David(300:770), Football1(1:74), Freeman3(1:460), Freeman4(1:283).

Basketball
Basketball   [zip]
IV, OCC, DEF, OPR, BC
Bolt
Bolt   [zip]
OCC, DEF, IPR, OPR
Boy
Boy   [zip]
SV, MB, FM, IPR, OPR
Car4
Car4   [zip]
IV, SV
CarDark
CarDark   [zip]
IV, BC
CarScale
CarScale   [zip]
SV, OCC, FM, IPR, OPR
Coke
Coke   [zip]
IV, OCC, FM, IPR, OPR, BC
Couple
Couple   [zip]
SV, DEF, FM, OPR, BC
Crossing
Crossing   [zip]
SV, DEF, FM, OPR, BC
David
David   [zip]
IV, SV, OCC, DEF, MB, IPR, OPR
David2
David2   [zip]
IPR, OPR
David3
David3   [zip]
OCC, DEF, OPR, BC
Deer
Deer   [zip]
MB, FM, IPR, BC, LR
Dog1
Dog1   [zip]
SV, IPR, OPR
Doll
Doll   [zip]
IV, SV, OCC, IPR, OPR
Dudek
Dudek   [zip]
SV, OCC, DEF, FM, IPR, OPR, OV, BC
FaceOcc1
FaceOcc1   [zip]
OCC
FaceOcc2
FaceOcc2   [zip]
IV, OCC, IPR, OPR
Fish
Fish   [zip]
IV
FleetFace
FleetFace   [zip]
SV, DEF, MB, FM, IPR, OPR
Football
Football   [zip]
OCC, IPR, OPR, BC
Football1
Football1   [zip]
IPR, OPR, BC
Freeman1
Freeman1   [zip]
SV, IPR, OPR
Freeman3
Freeman3   [zip]
SV, IPR, OPR
Freeman4
Freeman4   [zip]
SV, OCC, IPR, OPR
Girl
Girl   [zip]
SV, OCC, IPR, OPR
Ironman
Ironman   [zip]
IV, SV, OCC, MB, FM, IPR, OPR, OV, BC, LR
Jogging
Jogging   [zip]
OCC, DEF, OPR
Jumping
Jumping   [zip]
MB, FM
Lemming
Lemming   [zip]
IV, SV, OCC, FM, OPR, OV
Liquor
Liquor   [zip]
IV, SV, OCC, MB, FM, OPR, OV, BC
Matrix
Matrix   [zip]
IV, SV, OCC, FM, IPR, OPR, BC
Mhyang
Mhyang   [zip]
IV, DEF, OPR, BC
MotorRolling
MotorRolling   [zip]
IV, SV, MB, FM, IPR, BC, LR
MountainBike
MountainBike   [zip]
IPR, OPR, BC
Shaking
Shaking   [zip]
IV, SV, IPR, OPR, BC
Singer1
Singer1   [zip]
IV, SV, OCC, OPR
Singer2
Singer2   [zip]
IV, DEF, IPR, OPR, BC
Skating1
Skating1   [zip]
IV, SV, OCC, DEF, OPR, BC
Skiing
Skiing   [zip]
IV, SV, DEF, IPR, OPR
Soccer
Soccer   [zip]
IV, SV, OCC, MB, FM, IPR, OPR, BC
Subway
Subway   [zip]
OCC, DEF, BC
Suv
Suv   [zip]
OCC, IPR, OV
Sylvester
Sylvester   [zip]
IV, IPR, OPR
Tiger1
Tiger1   [zip]
IV, OCC, DEF, MB, FM, IPR, OPR
Tiger2
Tiger2   [zip]
IV, OCC, DEF, MB, FM, IPR, OPR, OV
Trellis
Trellis   [zip]
IV, SV, IPR, OPR, BC
Walking
Walking   [zip]
SV, OCC, DEF
Walking2
Walking2   [zip]
SV, OCC, LR
Woman
Woman   [zip]
IV, SV, OCC, DEF, MB, FM, OPR

Attributes

We have manually tagged the test sequences with 9 attributes, which represents the challenging aspects in visual tracking.

NAME DESCRIPTION
IV Illumination Variation – the illumination in the target region is significantly changed.
Basketball, Car4, CarDark, Coke, David, Doll, FaceOcc2, Fish, Ironman, Lemming, Liquor, Matrix, Mhyang, MotorRolling, Shaking, Singer1, Singer2, Skating1, Skiing, Soccer, Sylvester, Tiger1, Tiger2, Trellis, Woman
SV Scale Variation – the ratio of the bounding boxes of the first frame and the current frame is out of the range [1/ts, ts], ts > 1 (ts=2).
Boy, Car4, CarScale, Couple, Crossing, David, Dog1, Doll, Dudek, FleetFace, Freeman1, Freeman3, Freeman4, Girl, Ironman, Lemming, Liquor, Matrix, MotorRolling, Shaking, Singer1, Skating1, Skiing, Soccer, Trellis, Walking, Walking2, Woman
OCC Occlusion – the target is partially or fully occluded.
Basketball, Bolt, CarScale, Coke, David, David3, Doll, Dudek, FaceOcc1, FaceOcc2, Football, Freeman4, Girl, Ironman, Jogging.1, Jogging.2, Lemming, Liquor, Matrix, Singer1, Skating1, Soccer, Subway, Suv, Tiger1, Tiger2, Walking, Walking2, Woman
DEF Deformation – non-rigid object deformation.
Basketball, Bolt, Couple, Crossing, David, David3, Dudek, FleetFace, Jogging.1, Jogging.2, Mhyang, Singer2, Skating1, Skiing, Subway, Tiger1, Tiger2, Walking, Woman
MB Motion Blur – the target region is blurred due to the motion of target or camera.
Boy, David, Deer, FleetFace, Ironman, Jumping, Liquor, MotorRolling, Soccer, Tiger1, Tiger2, Woman
FM Fast Motion – the motion of the ground truth is larger than tm pixels (tm=20).
Boy, CarScale, Coke, Couple, Deer, Dudek, FleetFace, Ironman, Jumping, Lemming, Liquor, Matrix, MotorRolling, Soccer, Tiger1, Tiger2, Woman
IPR In-Plane Rotation – the target rotates in the image plane.
Bolt, Boy, CarScale, Coke, David, David2, Deer, Dog1, Doll, Dudek, FaceOcc2, FleetFace, Football, Football1, Freeman1, Freeman3, Freeman4, Girl, Ironman, Matrix, MotorRolling, MountainBike, Shaking, Singer2, Skiing, Soccer, Suv, Sylvester, Tiger1, Tiger2, Trellis
OPR Out-of-Plane Rotation – the target rotates out of the image plane.
Basketball, Bolt, Boy, CarScale, Coke, Couple, David, David2, David3, Dog1, Doll, Dudek, FaceOcc2, FleetFace, Football, Football1, Freeman1, Freeman3, Freeman4, Girl, Ironman, Jogging.1, Jogging.2, Lemming, Liquor, Matrix, Mhyang, MountainBike, Shaking, Singer1, Singer2, Skating1, Skiing, Soccer, Sylvester, Tiger1, Tiger2, Trellis, Woman
OV Out-of-View – some portion of the target leaves the view.
Dudek, Ironman, Lemming, Liquor, Suv, Tiger2
BC Background Clutters – the background near the target has the similar color or texture as the target.
Basketball, CarDark, Couple, Crossing, David3, Deer, Dudek, Football, Football1, Ironman, Liquor, Matrix, Mhyang, MotorRolling, MountainBike, Shaking, Singer2, Skating1, Soccer, Subway, Trellis
LR Low Resolution – the number of pixels inside the ground-truth bounding box is less than tr (tr =400).
Deer, Ironman, MotorRolling, Walking2

Visual trackers

We have tested 29 publicly available visual trackers. The trackers are listed in chronological order.

NAME CODE REFERENCE
CPF CPF P. Pe ́rez, C. Hue, J. Vermaak, and M. Gangnet. Color-Based Probabilistic Tracking. In ECCV, 2002.
KMS KMS D. Comaniciu, V. Ramesh, and P. Meer. Kernel-Based Object Tracking. PAMI, 25(5):564–577, 2003.
SMS SMS R. Collins. Mean-shift Blob Tracking through Scale Space. In CVPR, 2003.
VR-V VIVID/VR R. T. Collins, Y. Liu, and M. Leordeanu. Online Selection of Discriminative Tracking Features. PAMI, 27(10):1631–1643, 2005. [www]
* We also evaluated four other trackers included in the VIVID tracker suite. (PD-VRS-VMS-V, and TM-V).
Frag Frag A. Adam, E. Rivlin, and I. Shimshoni. Robust Fragments-based Tracking using the Integral Histogram. In CVPR, 2006. [www]
OAB OAB H. Grabner, M. Grabner, and H. Bischof. Real-Time Tracking via On-line Boosting. In BMVC, 2006. [www]
IVT IVT D. Ross, J. Lim, R.-S. Lin, and M.-H. Yang. Incremental Learning for Robust Visual Tracking. IJCV, 77(1):125–141, 2008. [www]
SemiT SBT H. Grabner, C. Leistner, and H. Bischof. Semi-supervised On-Line Boosting for Robust Tracking. In ECCV, 2008. [www]
MIL MIL B. Babenko, M.-H. Yang, and S. Belongie. Visual Tracking with Online Multiple Instance Learning. In CVPR, 2009. [www]
BSBT BSBT S. Stalder, H. Grabner, and L. van Gool. Beyond Semi-Supervised Tracking: Tracking Should Be as Simple as Detection, but not Simpler than Recognition. In ICCV Workshop, 2009. [www]
TLD TLD Z. Kalal, J. Matas, and K. Mikolajczyk. P-N Learning: Bootstrapping Binary Classifiers by Structural Constraints. In CVPR, 2010. [www]
VTD J. Kwon and K. M. Lee. Visual Tracking Decomposition. In CVPR, 2010. [www]
CXT CXT T. B. Dinh, N. Vo, and G. Medioni. Context Tracker: Exploring supporters and distracters in unconstrained environments. In CVPR, 2011. [www]
LSK LSK B. Liu, J. Huang, L. Yang, and C. Kulikowsk. Robust Tracking using Local Sparse Appearance Model and K-Selection. In CVPR, 2011. [www]
Struck Struck S. Hare, A. Saffari, and P. H. S. Torr. Struck: Structured Output Tracking with Kernels. In ICCV, 2011. [www]
VTS J. Kwon and K. M. Lee. Tracking by Sampling Trackers. In ICCV, 2011. [www]
ASLA ASLA X. Jia, H. Lu, and M.-H. Yang. Visual Tracking via Adaptive Structural Local Sparse Appearance Model. In CVPR, 2012. [www]
DFT DFT L. Sevilla-Lara and E. Learned-Miller. Distribution Fields for Tracking. In CVPR, 2012. [www]
L1APG L1APG C. Bao, Y. Wu, H. Ling, and H. Ji. Real Time Robust L1 Tracker Using Accelerated Proximal Gradient Approach. In CVPR, 2012. L1_Tracker">[www]
LOT LOT S. Oron, A. Bar-Hillel, D. Levi, and S. Avidan. Locally Orderless Tracking. In CVPR, 2012. [www]
MTT MTT T.Zhang, B. Ghanem,S. Liu,and N. Ahuja. Robust Visual Tracking via Multi-task Sparse Learning. In CVPR, 2012. [www]
ORIA ORIA Y. Wu, B. Shen, and H. Ling. Online Robust Image Alignment via Iterative Convex Optimization. In CVPR, 2012. [www]
SCM SCM W. Zhong, H. Lu, and M.-H. Yang. Robust Object Tracking via Sparsity-based Collaborative Model. In CVPR, 2012. [www]
CSK CSK F. Henriques, R. Caseiro, P. Martins, and J. Batista. Exploiting the Circulant Structure of Tracking-by-Detection with Kernels. In ECCV, 2012. [www]
CT CT K. Zhang, L. Zhang, and M.-H. Yang. Real-time Compressive Tracking. In ECCV, 2012. [www]

Test results

The success plots for all sequences and attributes are listed below.
For an overlap threshold (x-axis of the plot), the success ratio is the ratio of the frames whose tracked box has more overlap with the ground-truth box than the threshold.
The values in the brakets in the figures are the AUC (area under curve), each of which is the average of all success rates at different thresholds when the thresholds are evenly distributed.

SUCCESS PLOTS OPE SRE TRE
Overall
IV
Illumination Variation
SV
Scale Variation
OCC
Occlusion
DEF
Deformation
MB
Motion Blur
FM
Fast Motion
IPR
In-Plane Rotation
OPR
Out-of-Plane Rotation
OV
Out-of-View
BC
Background Clutters
LR
Low Resolution

Similarly a precision plot shows the ratio of successful frames whose tracker output is within the given threshold (x-axis of the plot, in pixels) from the ground-truth, measured by the center distance between bounding boxes.

PRECISION PLOTS OPE SRE TRE
Overall
IV
Illumination Variation
SV
Scale Variation
OCC
Occlusion
DEF
Deformation
MB
Motion Blur
FM
Fast Motion
IPR
In-Plane Rotation
OPR
Out-of-Plane Rotation
OV
Out-of-View
BC
Background Clutters
LR
Low Resolution
Locations of visitors to this page