Supported Workshop Dataset

Although authors may use any datasets they choose in their submissions, the organizers are also providing a dataset of ten Creative Commons license movies, particularly useful for the challenge track of this workshop. Full ground truth as annotated by human annotators will be released for six of these movies which can be used by systems as a training set.

If you make use of the dataset or otherwise participate in the workshop please cite this paper using the following bibtex:

@inproceedings{curtis2020hlvu,

  title={HLVU: A New Challenge to Test Deep Understanding of Movies the Way Humans do},

  author={Curtis, Keith and Awad, George and Rajput, Shahzad and Soboroff, Ian},

  booktitle={Proceedings of the 2020 International Conference on Multimedia Retrieval},

  pages={355--361},

  year={2020}

Movie Dataset

The 10 movie dataset (High-Level Video Understanding (HLVU)) is available from this link drive.google.com/drive/folders/1q1Ca0aFJrF9tB8hsw-mrI9d4tzy5wlPZ. This dataset comprises 10 movies with a Creative Commons license which can be used for submissions for this workshop. This dataset will be annotated by human assessors and ground truth (Ontology of relations, entities, actions & events, names and images of all main characters, & Knowledge Graph for the development/training (50%) movies). Full details of these movies are provided:

Honey - Romance - 86 mins.
Let's bring back Sophie - Drama - 50 mins.
Nuclear Family - Drama - 28 mins.
Shooters - Drama - 41 mins.
Spiritual Contact The Movie - Fantasy - 66 mins.
Super Hero - Fantasy - 18 mins.
The Adventures of Huckleberry Finn - Adventure - 106 mins.
The Big Something - Comedy - 101 mins.
Time Expired - Comedy / Drama - 92 mins.
Valkaama - Adventure - 93 mins.

Query types

Fill in the graph space: Fill in spaces in the Knowledge Graph (KG). Given the listed relationships, events or actions for certain nodes, where some nodes are replaced by variables X, Y, etc., solve for X, Y etc. Example of The Simpsons: X Married To Marge. X Friend Of Lenny. Y Volunteers at Church. Y Neighbor Of X. Solution for X and Y in that case would be: X = Homer, Y = Ned Flanders.
Relations between characters: How is character X related to character Y ? This query type question asks participants about all routes through the KG from one person to another. The main objective of this query type is to test the quality of the established KG. If the system managed to build a representative KG reflecting the real story line of the movie, then it should be able to return back all valid paths, including the shortest, between characters (i.e. how they are related to each other).

Metrics

Results will be treated as ranked list of result items per each unknown variable and the Reciprocal Rank score will be calculated per unknown variable and Mean Reciprocal Rank (MRR) per query.
In this query type, systems are asked to submit all valid paths from a source node to another target node with the goal of maximizing recall and precision. NIST will first evaluate whether each path is a valid path (i.e the submitted order of nodes and edges leads to a path from the source person to the target person of the query) and report the recall, precision and F1 measures.

Sample Queries and Responses

Relational paths between characters:

Sample Query:

<DeepVideoUnderstandingTopicQuery question="1" id="1">

  <item source="Superintendent Chalmers" target="Lenny"/>

  <item description="List all possible paths between Superintendent Chalmers and Lenny"/>

</DeepVideoUnderstandingTopicQuery>

Sample Response:

<DeepVideoUnderstandingTopicResult question="1" id="1" path="1">

  <item source="Superintendent Chalmers" relation="Superintendent At" target="Springfield Elementary"/>

  <item source="Springfield Elementary" relation="Studied At By" target="Bart"/>

  <item source="Bart" relation="Child_of" target="Homer"/>

  <item source="Homer" relation="Friend_of" target="Lenny"/>

</DeepVideoUnderstandingTopicResult>

Fill in the graph space (Optional):

Sample Query:

<DeepVideoUnderstandingTopicQuery question="2" id="1">

  <item subject="Person:Unknown_1" predicate="Relation:Spouse Of" object="Person:Marge"/>

  <item subject="Person:Unknown_1" predicate="Relation:Parent Of" object="Person:Bart"/>

  <item subject="Person:Unknown_1" predicate="Relation:Parent Of" object="Person:Lisa"/>

  <item subject="Person:Unknown_1" predicate="Relation:Friend Of" object="Person:Lenny"/>

  <item subject="Person:Unknown_1" predicate="Relation:Friend Of" object="Person:Barnie"/>

  <item subject="Person:Unknown_1" predicate="Relation:Socialises At" object="Entity:Moe's Tavern"/>

  <item subject="Person:Unknown_1" predicate="Relation:Works At" object="Entity:Nuclear Power Plant"/>

  <item subject="Person:Unknown_1" predicate="Relation:Attends" object="Entity:<BLANK>"/>

  <item description="Which Person has the following Relations: Spouse Of Person:Marge, Parent Of Person:Bart, Parent Of Person:Lisa, Friend Of Person:Lenny, Friend Of Person:Barnie, Socialises At Entity:Moe's Tavern, Works At Entity:Nuclear Power Plant, Attends Entity:<BLANK>?"/>

</DeepVideoUnderstandingTopicQuery>

<DeepVideoUnderstandingTopicQuery question="2" id="2">

  <item subject="Person:Unknown_2" predicate="Relation:Superintendent At" object="Entity:Springfield Elementary"/>

  <item subject="Person:Unknown_2" predicate="Relation:Supervisor Of" object="Person:Principal Skinner"/>

  <item subject="Person:Unknown_2" predicate="Relation:Attends" object="Entity:Church"/>

  <item description="Which Person has the following Relations: Superintendent At Entity:Springfield Elementary, Supervisor Of Person:Principal Skinner, Attends Entity:Church?"/>

</DeepVideoUnderstandingTopicQuery>

Sample Response:

<DeepVideoUnderstandingTopicResult question="2" id="1">

  <item order="1" subject="Homer" confidence="64"/>

  <item order="2" subject="Apu" confidence="18"/>

  <item order="3" subject="Flanders" confidence="12"/>

  <item order="4" subject="Reverend Lovejoy" confidence="6"/>

</DeepVideoUnderstandingTopicResult>

<DeepVideoUnderstandingTopicResult question="2" id="2">

  <item order="1" subject="Superintendent Chalmers" confidence="92"/>

  <item order="2" subject="Agnes Skinner" confidence="8"/>

</DeepVideoUnderstandingTopicResult>

Question Answering:

Sample Query:

<DeepVideoUnderstandingTopicQuery question="3" id="1">

  <item subject="Person:Ms. Krabappel" predicate="Relation:Unknown_1" object="Entity:Springfield Elementary"/>

  <item description="What is the relation / connection from Ms. Krabappel to Springfield Elementary?"/>

  <Answers>

    <item type="Entity" answer="Attends"/>

    <item type="Entity" answer="Teacher At"/>

    <item type="Entity" answer="Owns"/>

    <item type="Entity" answer="Studies At"/>

    <item type="Entity" answer="Principal At"/>

    <item type="Entity" answer="Friend Of"/>

  </Answers>

</DeepVideoUnderstandingTopicQuery>

Sample Response:

<DeepVideoUnderstandingTopicResult question="3" id="1">

  <item type="Relation" answer="Teacher_At"/>

</DeepVideoUnderstandingTopicResult>

Google Sites

Report abuse