Competition Phases

We are using Codalab as an evaluation engine. Following registration, a link to our evaluation microsite will be sent. You need to form a competition team to participate in the competition. Individuals are also encouraged to participate but they will need to create a team with one member. (More about Codalab and how to form teams.)

Training and Practice Phase

During the first phase of the competition, you can download the following files:

  • Training data: train.json

  • Unlabeled development (practice) data: dev.json

  • Unlabeled evaluation data: eval.json

After you develop your model using the training data, create a submission file dev.csv for the development data, as described in Submission Format. The name of the file must be dev.csv. Create a ZIP file that includes dev.csv. (Any name can be used for the ZIP file.) Note that the dev.csv file should be at the top level of the ZIP file's directory structure. (In other words, it should NOT be in any sub-directory in the ZIP file.)

You can submit your ZIP file by going to the Codalab microsite, choose Participate, then Submit/View Results. After the submit, refresh the page to see your score. It may take several minutes for the score to be posted on the leaderboard. The score for the development data is only used for practice and to verify your submission format.

Evaluation Phase

Similarly to the second phase of the competition, you can download the following files:

  • Training data: train.json (same as Round 1)

  • Unlabeled development (practice) data: dev.json (new)

  • Unlabeled evaluation data: eval.json (new)

Create a prediction file eval.csv for the evaluation data, as described in Submission Format. The name of the file must be eval.csv. Create a ZIP file that includes eval.csv. (Any name can be used for the ZIP file.) Note that the eval.csv file should be at the top level of the ZIP file's directory structure. (In other words, it should NOT be in any sub-directory in the ZIP file.)

You can submit your ZIP file by going to Participate, then Submit/View Results. In this phase, the leaderboard is hidden. You will not be able to see the performance of your model on the evaluation data until the competition is over.

Note that there is a limit on the number of submissions, and only the LAST submission of each team will be considered.