Shared Task: Text-Graph Representations for KGQA  

About

TextGraph-17 workshop co-located with the 62nd Annual Meeting of the Association for Computational Linguistics (ACL-2024) in Bangkok, Thailand on August 11–16, 2024 features a shared task presented at this page.   

TextGraph fosters investigation of synergies between methods for text and graph processing. This edition focuses on the fusion of LLMs with KGs. In line with this goal we propose a shared task on Text-Graph Representations for Knowledge Graph Question Answering (KGQA).  

The the shared task is to select a KG entity (out of several candidates) which correspond to an answer given a textual question. The specificity of the task, is that for each question-answer (Q-A) pair not only textual Q-A pair is given but also a graph of shortest paths in the KG from entities in query to the LLM-generated candidate entity (including links of the intermediate nodes). This way, participants easily may experiment with various strategies of text-graph modality fusion for the given task in a controllable manner.  

So the goal is learn how LLMs output can be enhanced with KGs. We propose a convenient testbed for it by pre-extracting the graph as there a many ways how this extraction can be done fragmenting the text-graph modality data fusion experiments. 

Important Dates

• Training dataset released: 10th March 2024

• Test set released: 25 March 2024

• End of evaluation: 25 April 6 May 2024

• Submission deadline for technical reports: 17 20 May 2024

Task Description

Participants are given 

• text: question with a list of Wikidata entities mentioned 

• text: 5-10 answer candidates in the form of Wikidata entities

• graph: a Wikidata sub-graph composed of the shortest paths between entities in question and entities in answer is provided

One of the candidates is correct, others are incorrect. The goal is to find the correct answer ie. perform a binary classification.  We make it easy to experiment with various text-graph modality fusion strategies by pre-extraction of both text and graph modalities.  

Evaluation metric: F1 score (the task is a binary classification task) 

Data and Code:

Participants are provided a train and development dataset in the form of Q-A-Subgraph triples.Besides, a submission to Codalab public and private breadboard of the test set Q-A solutions is available:

• Codalab: https://codalab.lisn.upsaclay.fr/competitions/18214 

• Github: https://github.com/uhh-lt/TextGraphs17-shared-task  

The illustration below shows the overall workflow of subgraph extraction and their use for text-graph based re-ranking of end-to-end LM answers (in our baselines the T5-ssm-nq model is used):

Examples of Training Data

Training data are represented in form of Questions and Answer with per-extracted candidates using an LLM and sub-graphs extracted from WikiData representing entities from question and answer as shown below. 

A key goal of the Text-Graph shared task is to learn about ways of combining textual and network representations. To make the experiment controllable, we pre-extract graphs (provided in the last column of each question in NetworkX format). Example of visualization for the question "Who was formerly an actor an now a Republican senator?" are presented below. Green node represent the correct answer, red node represent wrong answer candidate, blue node represent entities from the question, finally gray nodes are intermediate node on the shortest path from entities from question to answer. The participants are provided both textual labels of candidates and such graphs so additional features can be extracted from them, such as graph density, length of paths, textual labels on the paths, etc. More examples of graphs can be found online.

Instructions for Participants

1. Register to the shared task: https://forms.gle/HfKzqQ1PgkLUnKee9 

2. Download repository with evaluation script and data: https://github.com/uhh-lt/TextGraphs17-shared-task. 

   • Training dataset: pairs of questions and answer candidates together with graphs 

   • Evaluation script: can be used to locally test Q-A performance. 

   • Baseline solution: you can start from this simple version or use it as a template. 

   • Visuzliation script: can be used to visualize sub-graphs similarly as shown above. 

3. Submit your solution of test set to Codalab leadearboard: https://codalab.lisn.upsaclay.fr/competitions/18214. One submission is sufficient for both public and private leaderboard. Results of private set will be shown after the deadline.  

   • A sample submission corresponding to random baseline is available here: https://github.com/uhh-lt/TextGraphs17-shared-task/tree/main/submission_example 

4. Submit your technical report by the 20th of May.

   • Page limit is 4 pages + unlimited references + up to 2 pages of appendices.  

   • OpenReview link: https://openreview.net/group?id=aclweb.org/ACL/2024/Workshop/TextGraphs-17#tab-recent-activity 

   • Your paper title shall be following this template: "TEAMNAME at TextGraphs-17 Shared Task: PAPER TITLE." Please only change variables here TEAMNAME and PAPER TITLE. Here is an example of the report from the previous year. https://aclanthology.org/2020.textgraphs-1.12/ 

   • Papers must be anonymous 

   • Please cite paper about the shared task to provide the context to the readers: 

Results

@inproceedings{sakhovskiy-et-al-2024-textgraphs,

    title = "{T}ext{G}raphs 2024 Shared Task on Text-Graph Representations for Knowledge Graph Question Answering”,

    author = "Sakhovskiy, Andrey and Salnikov, Mikhail and Nikishina, Irina and Usmanova, Aida and Kraft, Angelie and Möller, Cedric and Banerjee, Debayan and Huang, Junbo and Jiang, Longquan and Abdullah, Rana and Yan, Xi and Ustalov, Dmitry and Tutubalina, Elena and Usbeck, Ricardo and Panchenko, Alexander",

    booktitle = "Proceedings of the TextGraphs-17: Graph-based Methods for Natural Language Processing",

    month = aug,

    year = "2024",

    address = "Bangkok, Thailand",

    publisher = "Association for Computational Linguistics",

  }

References

This section contains several relevant papers to the current shared task: 

• Sen, P., Fikri A. A., and Saffari, A. (2022): Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answering. In Proceedings of the 29th International Conference on Computational Linguistics, pages 1604–1619, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.

• Salnikov, M., Le, H., Rajput, P., Nikishina, I., Braslavski, P., Malykh, V.,  Panchenko, A. (2023): Large Language Models Meet Knowledge Graphs to Answer Factoid Questions. In Proceedings of the 37th Pacific Asia Conference on Language, Information and Computation (PACLIC 37). December 2–5. Hong Kong 

• Roberts, A,  Raffel, C., and Shazeer, N. (2020): How Much Knowledge Can You Pack Into the Parameters of a Language Model?. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418–5426, Online. Association for Computational Linguistics

Contact and Organizers

Please write all questions about the shared task to textgraphs17@googlegroups.com. Also you are invited join our Telegram group where also you can connect to organizers and get updates: https://t.me/+kRTCZYTrpJ5jZGVi   

Organizers:

• Irina Nikishina, Universität Hamburg

• Aida Usmanova, Leuphana University Lüneburg

• Angelie Kraft, Universität Hamburg

• Cedric Möller, Universität Hamburg

• Debayan Banerjee, Universität Hamburg

• Junbo Huang, Universität Hamburg

• Longquan Jiang, Universität Hamburg

• Rana Abdullah, Universität Hamburg

• Xi Yan, Universität Hamburg

• Andrey Sakhovskiy, KFU

• Elena Tutubalina, AIRI

• Mikhail Salnikov, AIRI 

• Alexander Panchenko, AIRI

• Ricardo Usbeck, Universität Hamburg

Acknowledgements

• Participants affiliated with Universität Hamburg extend their gratitude to NFDI4DS (National Research Data Infrastructure for Data Science) for their gracious support and endorsement of this shared task.