Data and evaluation
Evaluation
For the tracks, we split the data into training and testing partitions. For developing their methods, participants will use the training partition, and subsequently the test partition will be used to evaluate the participant methods and to determine the winner of the challenge.
Recommendation system evaluation
As in the 2021 edition, for the 2022 edition for the Recommendation system task, it will be evaluated with MAE (Mean Average Error) as in the Equation (1).
Where k is the k-th team to evaluate, n is the number of instances in the Test, GT is the Ground Truth, and S is the function that returns the instances of team k.
Sentiment analisys evaluation
For the sentiment analysis task for this edition, we propose two sub-tasks: the polarity classification and the type prediction. The polarity is a natural number in [1, 5]; this sub-task is evaluated with MAE. For the Type prediction, there are 3 classes (Attractive, Hotel, and Restaurant). For this reason, we apply the Macro F-measure as the Equation (2) indicates.
The final measure for this task is the average of the inverse of MAE and the Macro F1, as we can see in the Equation (3).
Semaphore prediction evaluation
For the last task, we propose to use the Macro F measure for the 4 sub-task (classify the weeks 0, 2, 4, and 8). Since we will give higher priority to correctly classify the weeks furthest away in time, in the case of this task, we propose a weighted average of macro F measures as the Equation (4) indicates.
Data
To access the data, you must register your team. Soon you will receive the data collection link.
Evaluation Rules
Runs for Track 1 will be received from 13th April 0:01 until 4th May 23:59 (-0600 UTC)
Runs for Track 2 will be received from 13th April 0:01 until 4th May, 23:59 (-0600 UTC)
Runs for Track 3 will be received from 13th April 0:01 until 4th May, 23:59 (-0600 UTC)
Participants are allowed to submit up to two runs for each track: one primary and one secondary. The participants must clearly flag each of the two.
Output Submission
Submissions formatted as described below and sent via email to the account: malvarez@cicese.edu.mx
Your software has to output for each task of the dataset a corresponding txt file. The file must contain one line per classified instance. Each line looks like this:
"TaskName"\t"IdentifierOfAnInstance"\t"Class"\n
It's important to respect the format with the " character, \t (tabulator) and \n (linux enter). The naming of the output files is up to you, we recommend to use the author and a run's identifier as filename with "txt" as extension.
For the recommendation system the possible labels are:
TaskName: recommendation
IdentifierOfAnInstance: NumberOfUser
Class: [1,5]
Output example:
" recommendation" "Usuario1" "2"
" recommendation" "Usuario2" "5"
" recommendation" "Usuario3" "3"
" recommendation" "Usuario4" "5"
" recommendation" "Usuario5" "1"
For the sentiment analysis the possible labels are:
TaskName: sentiment
IdentifierOfAnInstance:NumberOfOpinion
where NumberOfOpinion is the number line of the each opinion in the test file.
Classes: [1,5] '\t' [Attractive, Hotel, Restaurant]
Output example:
"sentiment" "1" "1" "Hotel"
"sentiment" "2" "2" "Hotel"
"sentiment" "3" "4" "Attractive"
"sentiment" "4" "3" "Hotel"
"sentiment" "5" "3" "Restaurant"
For the Semaphore prediction the labels are:
TaskName: semaphore
IdentifierOfAnInstance:NumberOfNews
where NumberOfNews is the number line of the each opinion in the test file.
Classes: week0Result '\t' week2Result '\t' week4Result '\t' week8Result
Where the possibles results are: [rojo, naranja, amarillo, verde]
Output example:
"semaphore" "1" "rojo" "rojo" "rojo" "naranja"
"semaphore" "2" "verde" "amarillo" "naranja" "rojo"
"semaphore" "3" "verde" "verde" "verde" "verde"
"semaphore" "4" "verde" "naranja" "naranja" "amarillo"
"semaphore" "5" ""amarillo" "verde" "verde" "rojo"
Notice that al instances number starts with 1.
A submission failing the format checking will be considered null.
Paper submission
Participants of the tasks will be given the opportunity to write a paper that describes their system, resources used, results, and analysis that will be part of the official IberLEF-2022 proceedings.
Here are some important considerations for the article:
System description papers should be formatted according to the Springer Conference Proceedings style: https://www.springer.com/gp/computer-science/lncs/conference-proceedings-guidelines. Latex and Word templates can be found there.
The minimum length of a regular paper should be 5 pages. There is no maximum page limit.
Papers must be written in English.
Each paper must include a copyright footnote on the first page of each paper: {\let\thefootnote\relax\footnotetext{Copyright \textcopyright\ 2022 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). IberLEF 2022, September 2022, Coruña, Spain.}}
Eliminate the numbering in the pages of the paper, if there is one, and make sure that there are no headers or footnotes, except the mandatory copyright as a footnote on the first page.
Authors should be described with their name and their full affiliation (university and country). Names must be complete (no initials), e.g. “Soto Pérez” instead of “S. Pérez”.
Titles of papers should be in emphatic capital English notation, i.e., "Filling an Author Agreement by Autocompletion" rather than "Filling an author agreement by autocompletion".
At least one author of each paper must sign the CEUR copyright agreement. Instructions and templates can be found at http://ceur-ws.org/HOWTOSUBMIT.html. The signed form must be sent along with the paper to the task organizers.