How do I register?
Register on the VAKRA-Advanced competition page: https://forms.gle/2MLpxaU1kcD1SbCm7. Registration is required for access to the evaluation dataset when it is released.
Does registration require me to submit?
No. You may register for evaluation data access and decide later whether to submit a system.
Can I participate alone or in a team?
Yes. Individuals and teams are welcome. There is no team size limit.
Must I enter all four subtasks?
No. You may enter any number of subtasks, including just one.
What are the subtasks?
API Chaining, Tool Selection, Multi-hop API Reasoning, and Multi-hop, Multi-source Reasoning. See the Subtasks & Data page for descriptions.
Where can I find the training dataset and environment?
The training dataset is available at Hugging Face. Code and environment setup instructions are on GitHub.
How do I access the evaluation dataset?
Register using the Google Form https://forms.gle/2MLpxaU1kcD1SbCm7. We will announce the release date and access instructions to registered participants.
When is evaluation?
Evaluation is planned for January 2027. We will announce the exact start and end dates.
What are the submission format and permitted resources?
Competition-specific submission instructions and rules on external resources will be published before the evaluation period.
Is the public VAKRA leaderboard the SemEval ranking?
No. The public benchmark leaderboard is separate from the official SemEval shared-task results.
Where will updates be posted?
Follow the task website and VAKRA-Advanced mailing list.
I cannot access the mailing list. Who can help?
Email ankita.naik@ibm.com and include the email address used to request access.