VAKRA-Advanced is a SemEval 2027 shared task on multi-hop, multi-source reasoning in realistic tool-grounded environments. Participants are expected to build agents that plan, select, and chain API calls while integrating information from structured tools and unstructured documents.
Four subtasks cover API Chaining, Tool Selection, Multi-hop API Reasoning, and Multi-hop, Multi-source Reasoning. Explore the Subtasks page for the task scope, then use the dataset and setup resources to get started.
The public VAKRA benchmark provides more than 8,000+ locally hosted APIs across 62 domains, backed by databases and aligned documents. It's executable environment and replayable traces support end-to-end assessment.
Register to access the evaluation dataset using the following form https://forms.gle/2MLpxaU1kcD1SbCm7