July 23rd -- The test phase is now complete. Thank you for your participation. Instructions below remain for posterity.

The test phase for the CRAFT Shared Tasks is now under way. The test data is available here.

Submissions will be accepted until July 22nd, 11:59pm IDLW (UTC−12:00).

You are permitted up to three (3) submissions for each task in which you are participating.

In order to submit results, you must register your team. Please only register once per team. Once registered, you will receive an email with a team ID and instructions for how to submit your results. Please note that the email you will receive in response to your registration is not an automated process. If you intend to participate, please register as soon as possible. While you will be able to submit results until the deadline listed above, please do not expect to be able to register near the deadline. Results will not be accepted after the final deadline. There will be no exceptions.

Submission guidelines

Result submissions will be made via a Google Form. The link to the form is provided to you when you register for the Shared Task. The form will prompt you to upload a file containing your results. Please only include relevant files in your results, i.e. only include files that are generated by your system. For example, files such as the ontologies used for the CA task should not be included in your results submission. Submission files should be packaged and compressed into either .tar.gz or .zip files. The expected directory structure and naming convention of the uploaded file is described below.

Submission naming convention

When you package your files for submission, please use the following naming convention for the uploaded file.

craft19-[TASK]-[TEAM_ID]-[RUN_ID].[SUFFIX]

where,

• [TASK] is an abbreviation for the task corresponding to this submission. It should be either CA, CR, or SA.
• [TEAM_ID] is the unique identifier for your team, provided in the email you received after registering, e.g. T999.
• [RUN_ID] is an identifier for the 'run' which you are submitting, e.g. run2. For each task you are permitted up to 3 separate submissions (runs). You will select the run identifier on the submission form. Please make sure it matches the run identifier on your uploaded file. In the event that the run identifer used in the file name and the run identifier selected in the google form do not match, the run identifier specified in the file name will be treated as the official run identifier.
• [SUFFIX] will depend on how you package your files and should be either .tar.gz or .zip.

Some example file names are the following:

• craft19-CA-T999-run1.tar.gz
• craft19-SA-T999-run3.zip

Submission directory structure

The directory structure of your uploaded file is task-specific, and should follow the expected evaluation directory structure as described for each specific task. For details, see the relevant wiki pages for the CA Task, CR Task, and SA Task.

Validate your submission

Before submitting results, please check for valid file formatting by evaluating your results against themselves using the appropriate CRAFT ST evaluation procedure. See instructions on the relevant wiki pages for the CA Task, CR Task, and SA Task.