Prediction submission

Evaluation schedule

The Seed Development evaluation will take place between July 9th and July 12th of 2019.

On July 9th 2019, we will publish:

• the untagged test set on the dataset page,
• the output of several pre-processings on the test set (tokenization, POS-tagging, parsing, term extraction) on the supporting resources page,
• a new version of the validation and evaluation software on GitHub.

We will accept submissions until the 12th of July 2019 23:59 AoE.

Shortly after we will publish the evaluation of all predictions on a dedicated page of the website.

Number of submissions

Each team is allowed three predictions (runs) for each sub-task.

Prediction format

Predictions must be sent in the form of a zip archive containing .a2 files in the BioNLP-ST file format.

The name of the archive must clearly indicate:

• The sub-task, either one of: SeeDev-binary, SeeDev-full.
• The number of the run: 1, 2 or 3.
• The name which you want your team identified with. It could be the name of your uni and lab, or the name of your software. Make it explicit, pronounceable, and unique. We will not evaluate anonymous submissions.

Checking your archive

Check your predictions before sending them using the provided validation and evaluation software, with the following command-line

java -jar bionlp-st-core-0.1.jar -task TASK -test -prediction YOURFILE.zip -check

• where TASK is the task identifier (SeeDev-binary, SeeDev-full)
• and YOURFILE.zip is the archive you intend to send.

Make sure to resolve all displayed error messages because non-conformity to the format and schema might put your submission in a disadvantage.

Licensing your predictions

We advise you to include a file named LICENSE in your archive to explicitly signal us what we can do with your predictions.

BioNLP-OST fosters openness and reproducibility, therefore we suggest the CC BY 4.0 license which gives access to the community while granting you moral rights.

We devised a template file to edit and include in the submitted archive. Replace the $TEAM, $SUBTASK and $OWNER placeholders. Make sure to set the $OWNER in conformance with the regulations of your country and employer.

Questionnaire

We kindly ask participants to fill the BioNLP-OST Participation Questionnaire in order to draw a landscape of shared task community. Individual answers will remain private, we will only publish statistics of the answers.

Send your submission

Submit your archive at the latest the 12th of July 2009 23:59 AoE by sending an e-mail to: Robert.Bossy@inra.fr. In the message title, make clear the name of the team and the subtask(s) you are submitting.

We will send you back a confirmation that we received your submission and a report of the validation software.

Participating to multiple sub-tasks and sending multiple runs

If you submit multiple runs, then prepare a separate archive for each run, each archive must be named so that the run number is clear. You may submit them through the same e-mail.

If you submit to several sub-tasks, then prepare a separate archive for each sub-task, each archive must be named so that the sub-task is clear. You may submit them through the same e-mail.