SOTA?

How to Participate?

To participate, please follow the following steps:

1. You should first sign up at the CLEF website: https://clef2024.imag.fr/index.php?page=Pages/registration.html . On the registration page, scroll down to "SimpleText - Improving Access to Scientific Texts for Everyone" and select "Task 4 - SOTA?." Please take a look at the other 3 tasks, i.e. Task 1, 2, and 3, offered by SimpleText https://simpletext-project.com/ and select one or more of those task options too if you are interested. Note that there is no interdependence of the dataset between "Task 4 - SOTA?" and the other three tasks of SimpleText.

2. Next, sign up on the codalab competition hosting site for SOTA?, i.e. Task 4 in the SimpleText track at CLEF 2024. We use codalab to host our competition, specifically, its evaluation and leaderboard. Participants will be required to submit their system submissions directly on codalab which will be automatically and uniformly evaluated. Please sign up per the following steps:

   1. Create an account on Codalab https://codalab.lisn.upsaclay.fr/. Sign in.

   2. Edit your profile appropriately. Make sure to add a team name, and enter names of team members. (Go to "Settings" and look under "Competition settings.")

   3. Proceed to the SOTA? task webpage on Codalab here https://codalab.lisn.upsaclay.fr/competitions/16616. Read information on all the pages (including Terms and Conditions). Click on the Participate tab and Register.

   4. Download the training data from Github here https://github.com/jd-coderepos/sota/.

   5. The Shared Task will be organized in two main evaluation phases, viz. Evaluation Phase 1: Few-shot Evaluation and Evaluation Phase 2: Zero-shot Evaluation. Please refer to our competitions timeline and our codalab competition site for detailed information. Our evaluation script is additionally released as open-source code on Github at the following link.

The SimpleText track at CLEF 2024 as a whole organizes 4 different tasks of which SOTA? is Task 4. For information on the whole track as well as the other tasks, please visit the SimpleText track webpage http://simpletext-project.com/2024/clef/ .

FAQ:

Q. If I am part of a team, can I make submissions to the task with my personal user account?

A. Yes and no. Yes, if as an individual person you are submitting your own system results that is different from the team system. No, if you are merely wishing to evaluate the team system with your individual account. We have a cap on the maximum allowed tries per system in each evaluation phase and doing this will violate that term. 

Q. How do I/we include more than one score on the leaderboard?

A. Codalab allows only one score on the leaderboard per user.

Contact

Feel free to ask any clarification questions on our task google groups at https://groups.google.com/u/5/g/sota-shared-task.