Summary of Previous Iteration of the Challenge

The 1st LLMs4OL challenge marked a notable direction toward employing LLMs in OL, demonstrating their potential in more automated knowledge acquisition. The challenge was structured into two evaluation phases: a few-shot phase, where models were trained on a subset of ontologies before being tested on related but unseen data, and a zero-shot phase, which assessed generalizability by introducing entirely new ontologies. Participants tackled three tasks: Term Typing (assigning types to terms), Taxonomy Discovery (identifying hierarchical relations), and Non-Taxonomic Relation Extraction (detecting semantic relationships beyond hierarchy). The challenge attracted eight teams of participants, each contributing novel systems for OL within the Semantic Web community’s broader goal of making the web more intelligent and user-friendly. Participants explored various strategies, including prompt engineering, fine-tuning, and hybrid models that combined LLMs with rule-based or retrieval-augmented approaches.

Overall, we observed that LLMs excel in Term Typing and Taxonomy Discovery, where their ability to infer hierarchical relationships and generalize across diverse knowledge domains proved beneficial. However, hybrid approaches consistently outperformed pure LLM methods in simpler tasks, indicating that incorporating external knowledge or structured reasoning enhances performance. Dataset diversity significantly impacted results, underscoring the need for well-curated benchmarks that ensure robustness and fairness across different ontologies and domains.

While different prompting and fine-tuning strategies led to varied success rates, Non-Taxonomic Relation Extraction remained a key challenge. LLMs struggled with capturing complex domain-specific relationships, particularly those requiring deeper semantic understanding beyond surface-level lexical cues. This suggests that current LLM architectures, while powerful in generalization, may still require additional structured learning techniques, specialized fine-tuning, or external knowledge retrieval to fully grasp relational semantics.

The challenge results highlight both the strengths and limitations of LLMs in OL. While they provide a promising avenue for reducing manual efforts in knowledge engineering, future work should focus on improving reasoning capabilities, developing domain-adaptive fine-tuning methods, and integrating LLMs with symbolic reasoning frameworks to enhance precision in complex OL tasks.

For further details, refer to the "The 1st LLMs4OL Challenge @ ISWC 2024 Proceeding".

Ontologies in LLMs4OL Challenge

The seen-eval evaluation ontologies are as follows (with their respective domain).

• Ontology for Biomedical Investigations (OBI) – Medicine – The OBI is a comprehensive, community-driven ontology that provides a structured framework for representing all aspects of biomedical and clinical investigations. It facilitates consistent annotation and integration of experimental data across diverse biomedical disciplines. 

• Material Ontology (MatOnto) – Material Science and Engineering –  The MatOnto is a domain-specific ontology designed to represent knowledge about materials, their properties, structures, and processing methods, primarily for use in materials science and engineering applications.

• Semantic Web for Earth and Environment Technology Ontology (SWEET) – Environment – The SWEET is an investigation in improving the discovery and use of Earth science data, through software understanding of the semantics of web resources. SWEET is a collection of ontologies conceptualizing a knowledge space for Earth system science and includes both orthogonal concepts (space, time, Earth realms, physical quantities, etc.) and integrative science knowledge concepts (phenomena, events, etc.).

• Human Disease Ontology (DOID) – Medicine – The Disease Ontology has been developed as a standardized ontology for human disease with the purpose of providing the biomedical community with consistent, reusable, and sustainable descriptions of human disease terms, phenotype characteristics, and related medical vocabulary disease concepts.

• Schema.org Ontology (SchemaOrg) – General Knowledge - Schema.org is a collaborative, community activity with a mission to create, maintain, and promote schemas for structured data on the Internet, on web pages, in email messages, and beyond.

• PROcess Chemistry Ontology (PROCO) – Chemistry – PROCO is a formal ontology that aims to standardly represent entities and relations among entities in the domain of process chemistry.

• Food Ontology (FoodON) – Agricultural – FoodOn, the food ontology, contains vocabulary for naming food materials and their anatomical and taxonomic origins, from raw harvested food to processed food products, for humans and domesticated animals. It provides a neutral and ontology-driven standard for government agencies, industry, nonprofits, and consumers to name and reference food products and their components throughout the food supply chain.

• Plant Ontology (PO) – Agricultural – The Plant Ontology (PO) is a structured vocabulary and database resource that links plant anatomy, morphology, and growth and development to plant genomics data.

• Gene Ontology (GO) – Biology and Life Sciences –The Gene Ontology (GO) provides structured controlled vocabularies for the annotation of gene products with respect to their molecular function, cellular component, and biological role.