Challenge Tasks

The EVAC 2024 challenge proposes the two following tasks for participants: 

Task 1 - Core affect presence and intensity

Participants will have to develop models for predicting affect presence and intensity of the core set labels. The affect presence prediction task involves binary classification, where the goal is to determine whether a specific affective label is present or absent for a given audiovisual sequence. A label is considered present if it was selected by at least one annotator, otherwise, it will be considered absent. The affect intensity prediction task is a regression task where the goal is to estimate the intensity of the affective labels of the core set. Labels of the core set include: confident, happy, interested, relaxed, satisfied, annoyed, anxious, desperate, frustrated, and surprised.

Task 2 - Appraisal dimensions

Participants will have to build models that predict dimensions which are provided either as a summary for each sequence, or a time-continuous series. In the dimensions summaries prediction task, participants focus on predicting summary values for each of the dimensions, which involves a regression task. The continuous dimension prediction task consists of predicting each of the dimensions over time with a time interval of 100 ms. Dimensions include: arousal, novelty, intrinsic pleasantness, goal conduciveness, and coping.

Guidelines

Participants will be provided with both training and development sets of subjects. The test partition will be released for a very short period of time at the end of the challenge for evaluating the systems. Whereas transcriptions of the verbal content is provided for the training and development partitions, transcriptions is not provided on the test partition. If applicable, participants will therefore need to perform automatic speech recognition on the test partition in order to exploit verbal information for the affect predictive tasks; organisers will compute and inform participants in this case of the Word Error Rate obtained on the transcriptions of the test partition.

Participants will be able to submit up to five different set of predictions on the test partition during the evaluation period. Organisers preserve the right to ask participants with the best two performing systems or in case of potentially dubious results. The outputs of the prediction models for the affective labels and dimensions will be compared with the gold-standard (i.e. references) using the Concordance Correlation Coefficient (CCC) as the evaluation metric.