Project 3: Ambiguity in Coreference

Coreference resolution is an important task in natural language processing. It requires language models to identify the correct antecedent for a particular mention. However, natural language is sometimes ambiguous, resulting in possibly more than one possible referent. Psycholinguistic studies have shown that some forms of anaphoric expressions are ambiguous, especially expressions that involve plural referents (Cokal et al., 2023; Koh & Clifton, 2002). For instances, in the sentence "The railway worker hooked up the engine to the goods wagon and sent it quickly to the central station". The word "it" can refer to either the engine or the good wagon and possibly to the entity formed by attaching the two objects. It is found that in such context, humans tend to use "it" to refer to the new entity formed by grouping the two components instead of one specific object. It is unknown whether LLMs are able to detect the ambiguity and identify the possible referents. In terms of overall performance, it has been shown that LLMs are not capable of modelling ambiguity (Liu et al., 2023). The goal of this PhD project is to mechanistically understand and direct the LLMs toward identifying ambiguity and possible referents of plural anaphoric expressions. In the literature, there is work attempting to build models for coreference resolution (Bohnet et al., 2023) and showing that LLMs are able to solve coreference resolution to some extent (Gan et al., 2024). However, there is very little work on how LLMs represent anaphoric referents in their internal workflow. Therefore, the project aims to (1) investigate the internal working of LLMs with respect to coreference resolution of singular and plural referents in both unambiguous and ambiguous cases and (2) develop models that can rigorously attend to ambiguity in plural anaphoric expressions. 

This PhD project is conducted by Thao Anh Dang - PhD Candidate at the Department of Information and Computing Science at Utrecht University.  Anh is supervised by Prof. Massimo Poesio and Prof. Rick Nouwen. For more information, please contact Anh at t.t.a.dang@uu.nl.

References

Bohnet, B., Alberti, C., & Collins, M. (2023). Coreference resolution through a seq2seq transition-based system. Transactions of the Association for Computational Linguistics, 11, 212-226. 

Cokal, D., Filik, R., Sturt, P., & Poesio, M. (2023). Anaphoric reference to mereological entities. Discourse Processes, 60(3), 202-223. 

Gan, Y., Poesio, M., & Yu, J. (2024). Assessing the Capabilities of Large Language Models in Coreference: An Evaluation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) (pp. 1645-1665). 

Koh, S., & Clifton Jr, C. (2002). Resolution of the antecedent of a plural pronoun: Ontological categories and predicate symmetry. Journal of Memory and Language, 46(4), 830-844. 

Liu, A., Wu, Z., Michael, J., Suhr, A., West, P., Koller, A., ... & Choi, Y. (2023). We’re Afraid Language Models Aren’t Modeling Ambiguity. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 790-807).