Evaluation

New evaluation period using updated datasets until May 6th, 2022, 00:00 UTC.


Training corpus

TRAINING DATA IS NOW AVAILABLE

Training data can be downloaded [HERE]

TEST DATA IS NOW AVAILABLE

Training data can be downloaded [HERE]

Also, you can download the test data on our [CODALAB PAGE]. Please register your team if you had not done it before.

Evaluation rules

  • The performance of your paraphrase identification solution will be ranked by the F1 measure on the paraphrase class.

  • Runs will be received from April 15th 0:01 until April 20th 23:59 UTC-12 (Anywhere on Earth)

  • Participants are allowed to submit up to two runs for each track: one primary and one secondary. The participants must clearly flag each of them.

CodaLab page

The CodaLab page will be available [here]. On this page, participants can evaluate their proposals on each phase of the task.

Output submission

Participants must send their results to the CodaLab page following the instructions described below.

The output for each track of the task must be a plain text file (.txt extension). This file must contain one line per classified instance. Each line needs the next format:

"TrackName"\t"IdentifierOfAnInstance"\t"Class"\n

It's important to respect the format with the " character, \t (tabulator) and \n (linux enter). The naming of the output files is up to you, we recommend to use the author and a run's identifier as filename with "txt" as extension.

For the Paraphrase identification task the possible labels are:

  • TrackName: ParaphraseIdentification

  • IdentifierOfAnInstance: NumberOfInstance

  • Class: {P, NP}

  • Output example:

"ParaphraseIdentification"\t"1"\"tP"\n

"ParaphraseIdentification"\t"2"\"tP"\n

"ParaphraseIdentification"\t"3"\"tNP"\n

"ParaphraseIdentification"\t"4"\"tNP"\n

"ParaphraseIdentification"\t"5"\"tP"\n


Paper submission

Participants of the tasks will be given the opportunity to write a paper that describes their system, resources used, results, and analysis that will be part of the official IberLEF-2022 proceedings. System description papers should be formatted according to the Springer Conference Proceedings style. Latex and Word templates can be found there. The minimum length of a regular paper should be 5 pages. There is no maximum page limit.

Papers must be written in English.


Sponsors

task.parmex@gmail.com