Movie Fill-in-the-Blank

Introduction

Question-answering has become a popular task, with many practical applications (e.g. dialogue systems). It's appealingly easy to interpret and quantitatively evaluate, and with a simple setup, it's relatively easy to generate the very large datasets that work well for deep learning. Advancements in visual QA (with images and natural language questions) with large-scale datasets have pushed this field forward rapidly, and with this challenge dataset we hope to extend this progress to video.

Challenge description

Given a video clip and a sentence with a blank in it, the task is to fill in the blank with the correct word.

Blanks are single-word only, from a vocabulary of about 3000 words including nouns, verbs, adjectives, and adverbs, with each blank occurring 50-3000 times in the training set. There are almost 300 000 training examples, from 100 000 clips. We provide a separate non-overlapping validation set with 21 000 examples (from 7500 clips), and evaluation will be on a test set of 30 000 examples (from 10 000 clips).

In the evaluation, we will also report performance by part-of-speech (nouns, verbs, adjectives, and adverbs), as well as human-defined categories based on visual information, to be announced soon (e.g. kissing/affection, anger/stress, fast motion, etc.). The intention of this evaluation is to examine how different models perform in different areas, as well as too look at how well our evaluation metrics correspond with intuitive categorizations.

Download

Data can be downloaded here.

Submission server

You can submit here.

Citations

Movie FiB paper: