2nd Workshop co-located with ESWC 2025, Portorož, Slovenia
Language models (LMs) have been considered promising in numerous knowledge engineering (KE) tasks, such as knowledge extraction, knowledge base construction, and curation. However, their adoption introduces new challenges for evaluation. The assessment of LM-generated results remains limited, lacking a comprehensive and formally defined framework, and relies heavily on human effort, making it difficult to compare methods and reproduce experiments.
The ELMKE workshop addresses this critical gap by spearheading a community-driven effort to standardize evaluation. It seeks to unite expertise, perspectives, and pioneering works to advance novel paradigms for evaluating LMs in KE. This workshop will showcase innovative and published papers focusing on evaluation methods for diverse KE tasks, such as completion and generative tasks. Additionally, discussions will explore challenges related to transparency, human evaluation, and broader reflections on the implications of evaluation methods. By establishing plans for platforms and dashboards for collaborative work, participants and the community can contribute to the design and implementation of robust evaluation methods and benchmarks, fostering targeted discussions and long-term collaboration.
Bio: Cogan Shimizu is an Assistant Professor at Wright State University in Dayton, Ohio, USA and is the director of the Knowledge and Semantic Technologies Laboratory. His work focuses on the construction of knowledge graphs and ontologies, especially with pattern-based methods, using both human-driven and AI-assisted methodologies. Other interests include knowledge engineering, broadly defined, from the use of knowledge representation (symbolic AI) into neural systems (e.g., Neurosymbolic AI) to the formulation of ontological understanding of domain science. His work has been published in the journal of web semantics, the semantic web journal, and various highly prominent Semantic Web conferences. Currently, he leads the Education Gateway to the Proto-OKN, which focuses on the development of educational material for knowledge graphs.
Abstract: Large Language Models bear the promise of significant acceleration of key Knowledge Graph and Ontology Engineering tasks, including ontology modeling, extension, modification, population, alignment, as well as entity disambiguation. We lay out LLM-based Knowledge Graph and Ontology Engineering as a new and coming area of research, and argue that modular approaches to ontologies will be of central importance.
Bio: Jennifer D’Souza is a senior postdoctoral researcher at the TIB Leibniz Information Centre for Science and Technology, where her expertise in AI, NLP, and scientific knowledge extraction drives her research. At TIB, Jennifer leads the NLP-AI aspect of the Open Research Knowledge Graph (ORKG) project and heads the SCINEXT project, aimed at advancing neuro-symbolic AI and NLP methods for scholarly innovation extraction, supported by the German Federal Ministry of Education and Research.
Abstract: In this talk, I will present our recent research in evaluating large language models (LLMs) for both structured and unstructured scientific summarization. Drawing from collaborative studies across the Open Research Knowledge Graph and scientific question answering benchmarks, my talk aims to highlight how LLMs can aid scalable, semantically rich synthesis while also addressing challenges in alignment, robustness, and evaluation reliability.
09:00 - 09:15
🎊 Welcome & Workshop Opening Session
09:15 - 10:30
[🎙️Keynote] Accelerating Knowledge Graph and Ontology Engineering with Large Language Models
Dr. Cogan Shimizu, Wright State University
10:30 - 11:00
☕ Coffee Break
11:00 - 11:45
[📃Presentation] Assessing the Capability of Large Language Models for Domain-Specific Ontology Generation
Anna Sofia Lippolis, Eva Blomqvist, Mohammad Javad Saeedizade, Andrea Giovanni Nuzzolese and Robin Keskisärkkä
11:45 - 12:30
[📃Presentation] From Experts to LLMs: Evaluating the Quality of Automatically Generated Ontologies
Majlinda Llugiqi, Fajar J Ekaputra and Marta Sabou
12:30 - 14:00
🥗 Lunch Break
14:00 - 15:30
[🎙️Keynote] Evaluating Large Language Models for Structured and Unstructured Science Summarization [Slides]
Dr. Jennifer D’Souza, TIB
15:30 - 16:00
☕ Coffee Break
16:00 - 16:45
[📃Presentation] How do Scaling Laws Apply to Knowledge Graph Engineering Tasks? The Impact of Model Size on Large Language Model Performance
Desiree Heim, Lars-Peter Meyer, Markus Schröder, Johannes Frey and Andreas Dengel
16:45 - 17:30
[📃Presentation] Large Language Models as Knowledge Evaluation Agents
George Hannah, Jacopo de Bernardinis, Terry Payne, Valentina Tamma, Andrew Mitchell, Ellen Piercy, Ewan Johnson, Andrew Ng, Harry Rostron and Boris Konev
17:30 - 18:00
💬 Discussion & Feedback Session
Mediteranea,
Floor 11,
Grand Hotel Bernardin,
Obala 2,
6320 Portorož,
Slovenia
Papers Submission Deadline: March 31, 2025
Papers Notification: April 18, 2025
Early Bird Registration: April 20, 2025
Papers Camera Ready: April 30, 2025
Workshop Day: June 2, 2025
All submission deadlines are 23:59:59 AoE (anywhere on earth).
Terry Payne, University of Liverpool
Nitisha Jain, King's College London
Daniel Dobriy, Vienna University of Economics and Business
Hamed Babaei Giglou, TIB
Chiara Di Bonaventura, King's College London
Ariana Graciotti, University of Bologna
George Hannah, University of Liverpool
Arunav Das, King's College London
Anna Sofia Lippolis, University of Bologna
Mohammad Javad Saeedizade, Linköping University
Yihang Zhao, King's College London