Participate
Participation Guideline in LLM4OL Challenge @ ISWC-2024
How to Participate?
You should first sign up at https://forms.gle/CkhLDpKwnWCztmJh6 and register your team. For organizational purposes, we are required to collect the list of participants.
We are currently running the training phase of the LLMs4OL 2024 Challenge. Please download the provided training dataset and proceed with training your systems.
If you download the UMLS dataset, it is mandatory to agree to the user agreement by UMLS. Please follow this link to complete the procedure. Note the UMLS dataset cannot be downloaded without complying with UMLS terms of use. This is a strict requirement.
To obtain the dataset, please visit https://www.nlm.nih.gov/databases/umls.html and follow the "How to Request a License and Create a UTS Account" section to obtain the License.
Once you have obtained the license, please submit a screenshot of the UMLS License to us via this form https://forms.gle/JgisYnQXjJixCgFZ9 or email us via llms4ol.challenge [at] gmail.com in order to receive the UMLS dataset for Tasks A, B, and C.
Join our Google group (https://groups.google.com/d/forum/llms4ol-challenge) and post your questions there. Refer to the timeline for when the training phase closes and the evaluation phase begins. All evaluations will be held on codalab. For this, the codalab evaluation page will be provided closer to the date when the evaluation phase begins.
FAQ
Q. If I am part of a team, can I make submissions to the task with my personal user account?
A. Yes and no. Yes, if as an individual person you are submitting your own system results that are different from the team system. No, if you are merely wishing to evaluate the team system with your individual account. We have a cap on the maximum allowed tries per system in each evaluation phase and doing this will violate that term.
Q. How do I/we include more than one score on the leaderboard?
A. Codalab allows only one score on the leaderboard per user.
Q. Can I participate in only one subtask?
A. Participation in all three tasks in the ISWC-LLMs4OL 2024 challenge is not mandatory. Thus participants can choose to enroll only in Task A or B or C subtasks, or Task A and B subtasks, or Task A and C subtasks, or Task B and C.
Q. Can I bring additional context from other sources?
A. Yes, additional context information from different sources such as the World Wide Web to enrich the training and test instances are welcome.
Contact
Feel free to ask any clarification questions on our challenge Google groups at https://groups.google.com/d/forum/llms4ol-challenge or contact us via llms4ol.challenge [at] gmail.com.