Abstract: In this talk I will briefly summarize the objectives and current state of research on meaning variation and disagreement, and will introduce the Dealing With Meaning Variation project and today's workshop.

Abstract: 

Human disagreement is ubiquitous and well known in labeling, yet variation in how people explain their judgments remains far less explored. 

This talk shows that human variation is not noise to be removed, but a valuable signal that language technologies can model and leverage for better alignment with diverse human perspectives. Recognizing that variation is holistic—shaping the entire NLP pipeline, from data collection to model training and evaluation—we address it at multiple stages. First, we show how models can be aligned with diverse perspectives through fine-tuning and in-context learning (ICL). Next, we rethink classification and explainability evaluation in light of variation. Finally, we outline future directions towards NLP systems designed around people’s needs, fostering pluralistic and trustworthy AI.

Abstract:

    Abstract: This talk presents the results of two studies in implicit discourse relation recognition (IDRR) with disagreement-aware methods based on the first crowdsourced corpus of implicit discourse relations, DiscoGeM (Scholman et al., 2022). The first study applies two families of disagreement-aware models, soft label models and perspectivist models, to IDRR and evaluates the advantages and drawbacks of each approach as well as the challenges involved. The second study attempts to disentangle errors and plausible interpretations in DiscoGeM and succeeds at diagnosing potentially misleading surface features in implicit relations, although struggling to detect straightforward errors like label spamming.

Abstract: In this talk, I will give an overview of my current research on human-gaze-integrated NLP. Human gaze is a rich behavioral signal: it reflects comprehension processes, reveals cognitive load, and serves as a communicative cue in situated interaction. My work explores how gaze data can be incorporated into NLP models to improve both their performance and interpretability. I will illustrate this through two downstream tasks -- multimodal reference resolution and hate speech detection -- the latter being a highly subjective task where grounding model decisions in human behavioral signals is particularly valuable. I will also discuss broader implications for modeling human label variation and subjectivity in annotation.

Abstract: In the HVar project (Disagreement in Corpus Annotation and Variation in Human Understanding of Text), we address human label variation in the annotation of textual coherence, focusing on coreference and discourse relations. The project builds on experience with inter-annotator disagreement in the Prague Dependency Treebank family and aims to identify and quantify systematic variation in text interpretation. Two corpora with multiple annotations and verbal justifications were created to investigate these phenomena: Hlava Cor focuses on coreference, while Hlava AD examines discourse relations. Initial analyses quantified inter-annotator agreement in terms of formal annotation decisions, such as the selection of antecedents or discourse arguments. Current research additionally examines annotators’ comments and their relationship to formal annotation decisions.The results show that formal agreement does not necessarily imply identical interpretations, while different formal solutions may sometimes reflect similar understandings. The comments also reveal linguistic factors affecting interpretation, including abstraction, syntactic complexity, and grammatical number. They further demonstrate individual differences among annotators, such as the influence of world knowledge and preferences for detailed or global reading. These findings raise methodological questions about how human label variation should be evaluated and suggest complementing multiple annotations with multiple evaluations and psycholinguistic approaches.

Abstract:  Coreference resolution is an important task in contextual reasoning. In this paper, we investigate the mechanism for representing and retrieving singular and plural entities for plural reference. We use a combination of mechanistic interpretability and attention pattern analysis to study the process in which LLMs predict a pronoun to refer back to previously mentioned entities. Using a range of causal intervention techniques, we find a set of attention heads that are responsible for (1) representing coreference information in the input, (2) identifying entities that form a plural reference, (3) transferring the information to the component that is responsible for selecting the antecedents and predicting the pronoun. We also find that LLMs align with humans in preference for plural pronoun. Specifically, entities in a plural construction are more likely to be referred to as a plural entity if they are ontologically similar and are linked by the conjunction "and".

Abstract:  Coreference is often ambiguous, and different annotators may provide equally valid interpretations of the same text. However, traditional evaluation metrics assume a single ground truth, potentially penalising systems that predict valid alternatives.

In this talk, I will present two complementary approaches to disagreement-sensitive coreference evaluation. The first extends existing coreference metrics by introducing virtual mentions and optimal bipartite alignment, allowing multiple valid annotations to be considered during evaluation. The second addresses large interpretation spaces arising from crowd-sourced annotations, using support-weighted evaluation to assess predictions without explicitly enumerating all possible interpretations.

We evaluate these approaches on Phrase Detectives 3.0 and the ARRAU RST test set, including an investigation of how large language models handle referential ambiguity. Our findings highlight the complementary strengths of the two methods and demonstrate how coreference evaluation can better reflect genuine variation in human interpretations.

Abstract: Dialogue agents are usually aligned to produce responses people prefer, but collaboration depends on whether participants actually hold compatible interpretations, and in asymmetric-information settings they can appear to coordinate smoothly while grounding the same expression to different referents. This talk presents work treating such friction (misalignment, clarification, repair) as an epistemic control signal within Frictive Policy Optimization. On the HCRC MapTask, we show that friction is only well-defined relative to each participant's information horizon: LLMs detect referential misalignment far better from a single participant's perspective than with omniscient access to both maps. We then operationalize Theory of Mind as a training signal through a four-part belief schema; ablating second-order beliefs cuts misunderstanding recall from 65% to 26%, and friction-conditioned objectives outperform DPO on intervention F1 and calibration while training far more stably. These results suggest that alignment for collaborative agents should be framed as regulating evolving belief relations: agents need to reason from a perspective and model what each party believes the other believes, so they can spot and repair silent misunderstandings before they compound. 

Abstract: In collaborative tasks with asymmetric information, participants coordinate their understanding through interaction. We ask whether gaze provides evidence about grounding across two such tasks. Working from discrete behavioral annotations, we map HCRC MapTask (Anderson et al., 1991) and MUNDEX (Türk et al., 2023) into a shared partner/task/away vocabulary and compute gaze features around task-relevant dialogue units. In both corpora, aligned reference interpretations (MapTask) and UND (understood) judgments (MUNDEX) are associated with more task-directed gaze and with less partner-directed gaze, lower gaze entropy, and fewer gaze transitions. The associations are clearest for the participant leading the task: in giver-produced references, and in explainer judgments, which also co-vary with the explainee's gaze. In same-speaker MapTask reference chains, the speaker's gaze entropy is lower at the mention where a previously non-aligned referent becomes aligned. The best gaze feature groups improve modestly over controls under grouped cross-validation: temporal features in MapTask and raw proportions in MUNDEX. Because effects are small and several weaken when recurring participants rather than dialogues are the unit of inference, we treat gaze as one contributing cue to grounding, to be interpreted alongside task and dialogue context.