Shared Task (Classification)

Note: The competition is now closed. Information is provided for reference only.

The Shared Task (Classification) consists of building a model based on the given datasets that will predict an emotion for each utterance in the evaluation dialogues.

The evaluation data will consist of two files with dialogues, one for Friends and one for EmotionPush. Due to the highly unbalanced nature of the data and a small number of three of the labels (surprise, disgust, and fear), you will only need to predict emotions using one of the following four labels: joy, sadness, anger, and neutral.

Please see below for details about the evaluation data and submission formats.

The predicted emotion will be compared to the gold labels. Utterances with gold labels which are different from the above four emotions will be discarded. For the final evaluation, we will compute the precision, recall and F1-scores for each of the four labels, and then summarize using micro F1-score.

Evaluation Data Format

The evaluation data is similar to the training data (i.e., an array of dialogues), but each dialogue line only includes the speaker and the utterance. The annotation and emotion fields are deleted.

Here is a example of the evaluation data file:

[
    [
        {
            "speaker": "Phoebe",
            "utterance": "Oh my God, he\u0092s lost it. He\u0092s totally lost it."
        },
        {
            "speaker": "Monica",
            "utterance": "What?"
        },
        ...
        ...
        ...
        {
            "speaker": "Chandler",
            "utterance": "You\u0092re a genius!"
        }
    ],
    [
        {
        ...
        ...
        }
    ]
    ...
    ...
]

Submission Format

As shown below, the submission file should be submitted in JSON format consisting of a top-level array which has two items. The first item is an object identifying the participant. The second object is the array of dialogues with labeled emotions (joy, sadness, anger, or neutral).

Make sure to replace your_name and your_email with your name and email.

[
    {
        "name": "your_name",
        "email": "your_email"
    },
    [
        [
            {
                "speaker": "Phoebe",
                "utterance": "Oh my God, he\u0092s lost it. He\u0092s totally lost it.",
                "emotion": "PREDICTED_EMOTION"
            },
            {
                "speaker": "Monica",
                "utterance": "What?",
                "emotion": "PREDICTED_EMOTION"
            },
            ...
            ...
            ...
            {
                "speaker": "Chandler",
                "utterance": "You\u0092re a genius!",
                "emotion": "PREDICTED_EMOTION"
            }
        ],
        [
            {
            ...
            ...
            }
        ]
        ...
        ...
    ]
]