Background

The Semantic Web aims to enrich the current web with structured knowledge and metadata for enhanced interoperability and understanding across systems. Central to this endeavor is Ontology Learning (OL), which automates the extraction of this structured knowledge from unstructured data, crucial for building dynamic ontologies foundational to the Semantic Web. The advent of Large Language Models (LLMs) has introduced a promising approach to OL, leveraging their deep linguistic understanding and pattern inference capabilities to automate OL. 

Our prior work published in the ISWC 2023 research track proceedings titled "LLMs4OL: Large Language Models for Ontology Learning" marked a notable direction towards employing LLMs in OL, demonstrating their potential in automating knowledge acquisition and representation for the Semantic Web. Based on this research, the LLMs4OL Challenge@ISWC-2024 as a community development endeavor in the 23rd ISWC call for challenges. With the ISWC-LLMs4OL 2024 challenge, we aim to catalyze community-wide engagement in validating and expanding the use of LLMs in OL. This initiative is poised to advance our comprehension of LLMs’ roles within the Semantic Web, encouraging innovation and collaboration in developing scalable and accurate ontology learning methods. 

This challenge aims to align with the Semantic Web community's goals of making the web more intelligent and user-friendly, offering a novel avenue for exploring the intersection of LLMs and OL. Participation in this challenge will contribute to evolving the Semantic Web, enabling more sophisticated services that utilize structured knowledge effectively.

There are two main evaluation phases for the challenge.

1. Few-shot testing phase 

Each ontology selected for system training will be divided into two parts: one part will be released for the training of the systems and another part will be reserved for the testing of systems in this phase. Furthermore, the evaluations will be conducted in two stages of consecutive time windows. In the first stage, only the test datasets for Task A will be released. Then the first stage evaluation window will be declared closed, i.e. no more submissions post the stage 1 evaluation period will be evaluated. Upon closing stage 1, the evaluations for stage 2 will commence. In stage 2, the test dataset for Tasks B and C will be released. This is deliberately planned to disallow for any new term types released in Task B or C test sets to influence systems' performance on the Task A test set.

2. Zero-shot testing phase

The zero-shot testing phase will also be organized into two stages that will run in parallel with the few-shot testing phase. The only difference is that new ontologies unseen during training will be introduced. The first stage will test for Task A and the subsequent stage will test for Tasks B and C. The zero-shot testing phase objective is to evaluate the generalizability and transferability of the LLMs developed in this challenge.   

NOTE: Ontologies offered for each of the tasks of the LLMs4OL challenge are publicly available sources. Participation in this task mandates that participants do not re-create new datasets from the publicly sourced ontologies as we reserve a portion of the respective ontologies for our few-shot testing phase. Thus participants are strictly restricted to using the dataset we as the challenge organizers release. Training on the re-created datasets from the full source ontologies would imply unfair testing of the systems where the systems would not be considered solutions to the OL problem. However, for our given datasets, participants are free to introduce context information from the World Wide Web, provided that this is explicitly stated in the supporting system submission documentation either in the publication or readme and would be considered part of the methodology.

Organizers