Challenges

In order to encourage the community to explore different modalities and build robust models for video understanding, we are hosting two video understanding challenges in our workshop that address different problems in the field. We believe that by presenting these complex challenges with recently released novel video datasets we can inspire a new generation of methods and architectures for modeling the highly complex tasks involved in multimodal video understanding.

We thank IBM for sponsoring the prize money of 5K!


Challenge A) Multimodal Multi-label Action Detection Challenge

This is a new challenge on multi-label action detection in video. For this, we use the recently introduced Moments in Time dataset with an expanded label set that includes multiple action labels per video. Labels in the dataset can pertain to actions recognizable using one or more of the audio/visual streams (i.e. audio actions, visual actions, or audio-visual actions).

Moments is an active research project in development by the MIT-IBM Watson AI Lab (http://moments.csail.mit.edu/). The project is dedicated to building a very large-scale dataset to help AI systems recognize and understand actions and events in videos. The dataset includes over 2 million action labels for over 800K 3 second videos.

Potential participants can visit the Moments in Time website to fill out a request form for download access to the dataset.

More information on the challenge can be found here.

Challenge results are released.

Challenge B) Temporal Action Localization Challenge

This is a temporal action localization challenge using the recently released HACS Segments dataset which has 140K action segments with temporal annotations from 50K untrimmed videos. This dataset includes actions such as “playing piano” and “gargling mouthwash” which can be better localized with the multimodal fusion of visual, audio and temporal information.

Please visit (http://hacs.csail.mit.edu/challenge.html) for detailed challenge information.


Important Dates:

  • July 5, 2019 : Training data and development kit released
  • August 1, 2019 : Testing data released
  • October 19, 2019 : Submission deadline
  • November 2, 2019 : Challenge winners present at workshop