LLMs4OL 2026: Large Language Models for Ontology Learning
The 3rd LLMs4OL Challenge @ ISWC 2026
ISWC 2026, Bari, Italy | 25-29 October
Embedded Files
Flagship Task
Flagship Task
End-to-End Ontology Learning
Definition: Given raw text, construct a primitive ontology including terms, types, taxonomy, and relations.
Motivation: Real-world ontology construction is not a collection of independent subtasks. It requires joint reasoning across: term/type discovery, term typing, axonomy discovery, and non-taxonomic relationship extraction. This task evaluates whether LLM-based systems can move from component intelligence to pipeline intelligence.
Objective
Objective
Given raw domain text, participants must construct a structured ontology by integrating all major Ontology Learning (OL) stages into a single, coherent pipeline. Unlike prior OL benchmarks that isolate subtasks (e.g., only taxonomy extraction), this flagship task evaluates how effectively systems—especially LLM-augmented systems—can compose multiple OL stages into an end-to-end ontology construction workflow.
What must participants build?
What must participants build?
A system that, starting from unstructured text, produces a primitive ontology containing:
Types (concepts or classes)
Terms (instances)
Term Typings (mappend instances to classes)
Taxonomic Discovery (is-a/subclass)
Non-taxonomic relations
A connected, coherent ontology graph
All stages must be automatically derived from text via an integrated pipeline.
A toy example
A toy example
Provided raw text:
In a smart home system, sensors monitor environmental conditions. A temperature sensor measures room temperature. A motion sensor detects movement and triggers the alarm system. The smart thermostat receives temperature readings from the temperature sensor and adjusts the heating system. A mobile app allows the user to control the smart thermostat remotely.
➡️ 1. Term Extraction
temperature sensormotion sensoralarm systemsmart thermostatheating systemmobile appuserroom temperaturetemperature readingmovement
➡️ 2. Types Extraction
devicesensorsystemapplicationenvironmental conditionmeasurementperson
➡️ 3. Term Typing
temperature sensor ---> sensormotion sensor ---> sensoralarm system ---> systemsmart thermostat ---> deviceheating system ---> systemmobile app ---> applicationuser ---> personroom temperature ---> environmental conditiontemperature reading ---> measurementmovement ---> environmental condition
➡️ 4. Taxonomic Discovery
(sensor, is-a, device)(system, is-a, device)(application, is-a, device)
➡️ 5. Non-Taxonomic RE
(temperature sensor, measures, room temperature)(motion sensor, detects, movement)(motion sensor, triggers, alarm system)(smart thermostat, receives, temperature reading)(smart thermostat, controls , heating system)(mobile app, controls, smart thermostat)(user, uses , mobile app)(sensor , monitors, environmental condition)
The final output would be as follows:
(temperature sensor, instance-of, sensor)
(motion sensor, instance-of sensor)
(alarm system, instance-of, system)
(smart thermostat, instance-of device)
(heating system, instance-of, system)
(mobile app, instance-of, application)
(user, instance-of, person)
(sensor, is-a, device)
(system, is-a, device)
(application, is-a, device)
(temperature sensor, measures, room temperature)
(motion sensor, detects, movement)
(motion sensor, triggers, alarm system)
(smart thermostat, receives, temperature reading)
(smart thermostat, controls, heating system)
(mobile app, controls, smart thermostat)
(user, uses, mobile app)
(sensor, monitors, environmental condition)
Dataset
Dataset
More information on train/test datasets (formatting) will be available soon! Stay tuned!
Evaluation Metrics
Evaluation Metrics
Standard Metrics: Precision, Recall, F1
Standard Task-specific Metrics: Precision, Recall, F1 scores, for Term Typing, Taxonomy Discovery, and Non-Taxonomic RE.
Graph Similarity Metric
Page updated
Google Sites
Report abuse