Task Rules

FAQ

1. Maximum Submission : 3 result submissions per subtask

2. Ranking: Two ranking per subtask - Per Language Ranking and Global Ranking

3. What results will be displayed/used in the LeaderBoard: All the measures given in baseline script (Precision, Recall, F1 and F1-macro for subtask1 and Rho for subtask2) will be shown, but the final ranking will be based on macro F1 and Rho.

4. The naming convention for submission file: The result/submission file will be tab separated (with headers: ID \t Labels/Score), named as answer.tsv and then compressed to a zip file with naming convention: <teamName_subtaskX_submissionNo.zip>, X={1,2} and No={1,2,3}

5. Results selected to display in Leaderboard: Each team will have 3 chances (per task) and from there they can choose which results to submit in the leaderboard. However, each team must submit at least one result in the board (they can change the selected entry to show anytime during competition). This is mainly given so participants attempting just selected language are not penalized by the global-ranking score mechanism.

Shared Task Rules:

Each subtask includes test data in three languages: English, French and Italian.

For each sub-task, the ranking will be two fold:

1. Ranking per language (En, Fr, It)

2. Global ranking - the average of the score from all the three languages (This will be the official ranking).

Each team can submit at max 3 results files for the evaluation.

The evaluation phase will be active for 5 days, then the portal will be closed. For the official ranking, we will use macro F1 (subtask 1) and Spearman correlation, Rho (subtask 2).

The participant team will submit one file containing the id and the corresponding prediction of the task. The submitted file will/can contain instances from (all) the languages. For example: if a team (team A) chooses to participate in both English and French-Subtask 1, then the submitted prediction file will contain instances from both languages. In this case, we will calculate the global ranking score (GRS) as follows:

GRS = (F1_macro (English) + F1_macro (French) + F1_macro (Italian))/3

So in this case, for team A, GRS will be calculated using F1_macro for English and French, while for Italian (not done) the F1_macro will be set to 0.

The same will be applied for SubTask 2.

The test data (from 18th January) is available in: https://github.com/shammur/SemEval2022Task3/

Here below is the format for the uploaded prediction file. More details on format are given in the git repository (data/test/sample_data)

Submission Format:

The participant team will submit a zip file containing the prediction tsv (named as answer.tsv [MUST!!]), the naming convention of the zip file will be as follows:

teamName_subtask1_submissionNo.zip (or teamName_subtask2_submissionNo.zip)

Containing → answer.tsv file (this file will include all/chosen language predictions.

SubTask 1:

Filename to be submitted: <answer.tsv> -> compressed to create <teamName_subtask1_submissionNo.zip>

#Tab separated: ID Labels

example: answer.tsv

ID Labels

en_2376 1

en_4481 0

en_3104 0

fr_2301 1

fr_1309 0

fr_1360 0

it_2625 0

it_3069 1

it_2414 0

SubTask2:

Filename to be submitted: <answer.tsv> -> compressed to create <teamName_subtask2_submissionNo.zip>

example: answer.tsv

ID Score

en_261 1.09

en_440 5.64

fr_19 6

fr_254 6.67

it_21 4.42

it_191 1.33