Shared Task: Text-Graph Representations for KGQA  

About

TextGraph-17 workshop co-located with the 62nd Annual Meeting of the Association for Computational Linguistics (ACL-2024) in Bangkok, Thailand on August 11–16, 2024 features a shared task presented at this page.   

TextGraph fosters investigation of synergies between methods for text and graph processing. This edition focuses on the fusion of LLMs with KGs. In line with this goal we propose a shared task on Text-Graph Representations for Knowledge Graph Question Answering (KGQA).  


The the shared task is to select a KG entity (out of several candidates) which correspond to an answer given a textual question. The specificity of the task, is that for each question-answer (Q-A) pair not only textual Q-A pair is given but also a graph of shortest paths in the KG from entities in query to the LLM-generated candidate entity (including links of the intermediate nodes). This way, participants easily may experiment with various strategies of text-graph modality fusion for the given task in a controllable manner.  

So the goal is learn how LLMs output can be enhanced with KGs. We propose a convenient testbed for it by pre-extracting the graph as there a many ways how this extraction can be done fragmenting the text-graph modality data fusion experiments. 

Important Dates

Task Description

Participants are given 

One of the candidates is correct, others are incorrect. The goal is to find the correct answer ie. perform a binary classification.  We make it easy to experiment with various text-graph modality fusion strategies by pre-extraction of both text and graph modalities.  

Evaluation metric: F1 score (the task is a binary classification task) 

Data and Code:

Participants are provided a train and development dataset in the form of Q-A-Subgraph triples.Besides, a submission to Codalab public and private breadboard of the test set Q-A solutions is available:

The illustration below shows the overall workflow of subgraph extraction and their use for text-graph based re-ranking of end-to-end LM answers (in our baselines the T5-ssm-nq model is used):

Examples of Training Data

Training data are represented in form of Questions and Answer with per-extracted candidates using an LLM and sub-graphs extracted from WikiData representing entities from question and answer as shown below. 

TextGraphs - Sample

A key goal of the Text-Graph shared task is to learn about ways of combining textual and network representations. To make the experiment controllable, we pre-extract graphs (provided in the last column of each question in NetworkX format). Example of visualization for the question "Who was formerly an actor an now a Republican senator?" are presented below. Green node represent the correct answer, red node represent wrong answer candidate, blue node represent entities from the question, finally gray nodes are intermediate node on the shortest path from entities from question to answer. The participants are provided both textual labels of candidates and such graphs so additional features can be extracted from them, such as graph density, length of paths, textual labels on the paths, etc. More examples of graphs can be found online.


Instructions for Participants




@inproceedings{sakhovskiy-et-al-2024-textgraphs,

    title = "{T}ext{G}raphs 2024 Shared Task on Text-Graph Representations for Knowledge Graph Question Answering”,

    author = "Sakhovskiy, Andrey and Salnikov, Mikhail and Nikishina, Irina and Usmanova, Aida and Kraft, Angelie and Möller, Cedric and Banerjee, Debayan and Huang, Junbo and Jiang, Longquan and Abdullah, Rana and Yan, Xi and Ustalov, Dmitry and Tutubalina, Elena and Usbeck, Ricardo and Panchenko, Alexander",

    booktitle = "Proceedings of the TextGraphs-17: Graph-based Methods for Natural Language Processing",

    month = aug,

    year = "2024",

    address = "Bangkok, Thailand",

    publisher = "Association for Computational Linguistics",

  }

References

This section contains several relevant papers to the current shared task: 

Contact and Organizers

Please write all questions about the shared task to textgraphs17@googlegroups.com. Also you are invited join our Telegram group where also you can connect to organizers and get updates: https://t.me/+kRTCZYTrpJ5jZGVi   


Organizers:

Acknowledgements