We encourage participants to focus on building and refining their systems during the solution submission phase and the integration during the system integration phase.

Solution Submission Phase

Based on our experience from previous editions of the challenge, we observed that leaderboard-driven platforms can sometimes encourage participants to focus primarily on incremental score improvements rather than on the broader scientific and methodological goals of the challenge. To foster deeper analysis, reproducibility, and methodological innovation, we have decided to adopt a simplified submission process this year.

Therefore, the challenge will not use CodaBench or a public leaderboard. Instead, participants will submit their predictions through a dedicated Google Form.

Test Sets:

The test sets for tasks are as follows.  

• Task A:  The Flagship Task test case is available at https://github.com/sciknoworg/LLMs4OL-Challenge/tree/main/2026/TaskA-Flagship (2018 test case)

• Task B: The Reuse Task test case is available at: https://github.com/sciknoworg/LLMs4OL-Challenge/tree/main/2026/TaskB-Reuse  (1799 test case)

• Task C: The Taxonomy Learning consists of multiple test cases, and the test cases are available at https://github.com/sciknoworg/LLMs4OL-Challenge/tree/main/2026/TaskC-Taxonomy. We have 4 test cases named as Phyra, Archeon, Biora, and Hylex. The prediction should be submitted for each test case separately.

Expected Output Formats:

For tasks, we expect strict formatting to ease the in-house evaluation and avoid any errors, so please take them seriously into consideration:

• Task A: For this task, we expect the following JSON format - a list of your system predictions, where each test case should be formatted as follows:

[

{

"id": "..." ,                                                        # ID of the test case - the same as the input sample id

"primitive-ontology-triples": [[subject, predicate, object], ...]    # List of triplets

   },

...

]

• Task B: Similar to Task A, we expect the following JSON format, but with the following key and values.

[

{

"id": "..." ,                                                                 # ID of the test case - the same as the input sample id

"extended-primitive-ontology-triples": [[subject, predicate, object], ...]    # List of new triplets 

   },

...

]

• Task C:   For this task, we expect the following JSON format - a list of your system predictions, which consist of parent and child pairs and should be formatted as follows:

[

{

"parent": "..." , 

"child": "..."    

   },

...

]

• Task D [Optional Ontologizer Task]: For submissions to this task, please add your final merge request URL to the OntoLearner in the submission form only your paper.

  •  [New] There will be no need to make any submission during the evaluation phases. For this task, we only need a short paper to be submitted with the paper submission.

  •  [New]  Please note that the paper submission for Task D should be isolated from other task papers. Which means if you are participating in Task A and D, you should submit one paper for Task A and another for Task D. OR if you are participating in Tasks A, B, C, and D, you should still submit two papers, one for Tasks A, B, and C, another for Task D. 

  •  [New]  We are accepting short papers for Task D.

Submission Policy

• Participants may submit up to five (5) runs per task until 17 June as an intermediate evaluation phase.

• A single intermediate evaluation will be conducted, and the results will be communicated privately to each participating team. This feedback is intended to help teams assess their approach and make improvements before the final submission.

• After 17 June, each team will be allowed to submit two (2) final runs for the official evaluation.

• No public rankings or leaderboards will be displayed during the challenge evaluation phase.

• The test set will remain unchanged throughout the challenge period. The intermediate evaluation and the final evaluation will be performed on the same test cases.

• Teams joining after 17 June will not be eligible for the intermediate evaluation feedback.

We believe this format will encourage participants to focus on developing robust and generalizable solutions while still providing an opportunity to receive feedback and refine their systems before the final evaluation.

Reporting Potential Issues with Test Cases

If you identify any potential issues with the test cases, please feel free to report them by providing the task name and the corresponding test case ID. We will carefully review each report and verify whether the test case contains an error or inconsistency.

While we have made every effort to ensure the quality and correctness of all test cases, minor issues may still remain. We greatly appreciate your feedback and collaboration in helping us maintain the highest possible quality of the evaluation data.

System Integration Phase 

Please note that participation in Phase 2 is mandatory for teams participating in Phase 1.

In this phase, the focus shifts toward system integration and finalization. The process will be kept simple to reduce overhead and support easy submission of working systems. Participants will be able to submit their system either as a single notebook or script, or via an OntoLearner issue, depending on what is most convenient. The goal of this phase is to ensure smooth integration of the developed approaches rather than introducing additional complexity. 

[New] How to submit the system integration phase?

The objective of this phase is to integrate and submit your contributions to OntoLearner. Contributors may choose one of the following submission formats.

Option 1: Jupyter Notebook Submission

If your implementation is notebook-based (or if you are willing to make it in this format), submit a single Jupyter Notebook containing your complete system. If multiple system variants share the same implementation but differ only in configuration, we recommend consolidating them into a single notebook. Your notebook should include the following sections:

1. Environment Setup:

• Include an installation section at the beginning of the notebook.

• Provide all required package installations using pip install commands.

• The notebook should execute successfully without requiring any additional dependencies.

2. Modular Implementation

• Organize the notebook into clearly defined sections, with each section representing a major component or stage of your system.

• Introduce each section with a brief Markdown description explaining its purpose and how it works (in short sentences).

• Place the corresponding implementation directly below the description at single or mutiple cells.

• Add inline comments where necessary to clarify non-trivial implementation details. Avoid excessive commenting of self-explanatory code.

3. Model Execution

• Include a dedicated section demonstrating how to execute your system.

• Provide a single execution cell that serves as the primary entry point.

• If your system supports multiple configurations or variants, implement separate main functions for each configuration and invoke the appropriate one.

• Each entry function should accept the required input paths, such as:

  • Test dataset path for inference.

  • Training dataset path if fine-tuning or training is supported.

4. Unit Testing

• Include a final section containing unit tests for the core components of your system.

• The tests should verify the functionality of the primary modules required for correct system execution.

5. Submission: After preparing the notebook, create an issue in the OntoLearner GitHub repository and specify your team name and challenge variant in the issue title. example TEAMA at LLMs4OL-2025

Option 2: Script-Based Submission

If your implementation consists of multiple Python scripts, please follow the guidelines below.

• Project Structure: Do not use nested directories or subdirectories. Please place all source files directly in the project root directory. For example:

project/

├── main.py

├── module1.py

├── module2.py

├── ....py

├── requirements.txt

├── README.md

└── unittest.py

• Entry Point: Provide a main.py file as the primary entry point for your system. If your submission includes multiple variants, provide separate entry files (e.g., main1.py, main2.py, etc.). Moreover, document the purpose of each entry point in the README.md.

• Requirements: Include a requirements.txt file listing all required dependencies.

• Documentation: Provide a README.md containing: 1) Instructions for running the system; 2) A brief description (one or two sentences) of each script and its purpose.

• Unit Testing: Include a unittest.py file containing unit tests for the critical modules of your implementation.

• Submission: Package the project as a ZIP archive and submit it by creating an issue in the OntoLearner GitHub repository. Your submission should clearly specify your team name and challenge variant.  Example TEAMA at LLMs4OL-2025

Option 3: Native OntoLearner Integration

Direct integration into the OntoLearner framework is optional. We understand that this approach may require additional effort and development time. If you choose this option:

1. Fork the OntoLearner repository.

2. Follow the OntoLearner guide page and make your system integration

3. Follow the commit guidelines when making a commit during your system integration.

4. Add the read-the-docs documentation page of your system.

5. Add unit tests for your system components.

6. Add an example run to the example directory of OntoLearner.

7. Submit your implementation as a Pull Request for review and be available during the review process so we can get your system into the Ontolearner.

8. Include your team name and challenge variant in the Pull Request title.

NOTE: In all forms of submissions, we are recommending to also include a methodological diagram of yours and your project GitHub repository so we could add these informations in the documentation websites.

Example integrations:

• https://ontolearner.readthedocs.io/learners/llms4ol_challenge/rwthdbis_learner.html

• https://ontolearner.readthedocs.io/learners/llms4ol_challenge/alexbek_learner.html

• https://ontolearner.readthedocs.io/learners/llms4ol_challenge/sbunlp_learner.html

• https://ontolearner.readthedocs.io/learners/llms4ol_challenge/skhnlp_learner.html

Submission Form

• The submission form can be accessed via: https://forms.gle/cgLWLhj6zzZTTayj6 

• The evaluation scripts are avaliable at: https://github.com/sciknoworg/LLMs4OL-Challenge/tree/main/2026/metrics