Pragmatic Reasoning in Language Models

Workshop at COLM 2025

October 10, 2025 | Montreal, Canada

IMPORTANT INFORMATION

📍Location: Room 520B

🔓 Submissions: Closed

❗ Submission deadline: June 27th AoE June 23rd AoE

✔️ Notification of acceptance: July 24th

📃 Camera-ready due: September 21st

🥳 Workshop date: October 10th

SUBMIT NOW

Join us at COLM:

WORKSHOP OVERVIEW

Humans are Pragmatic Language Users

We produce language based on our understanding of how context contributes to meaning and deliberate on the choice of utterances and interpretations that helps us collaborate and engage in social interactions. While recent large language models (LLMs) have shown impressive performance on a variety of language-related tasks, could these models be considered as true pragmatic language users?

Towards Language Models as Language Users

The 1st Workshop on Pragmatic Reasoning in Language Models (PragLM) aims to stimulate research on LLMs as pragmatically competent language users. We invite contributions that will forward the discussion of understanding and improvement of LLMs' capability to generate natural language flexibly and efficiently across contexts, with relations to research on the cognitive and linguistic processes supporting effective, context-sensitive communication. Our interdisciplinary theme brings together researchers in NLP, computational pragmatics, cognitive science, and other fields.

CALL FOR PAPERS

TOPICS OF INTEREST

We invite researchers to present their published and ongoing works on the topics of, but not limited to:

Improving Pragmatic, Contextual Language Use in LLMs
How can LLMs' pragmatic abilities be improved and made more human-like, and how should human-likeness in pragmatic competence be assessed?
Evaluating Pragmatic Competence of Language Models
How well can LLMs comprehend and/or generate pragmatic language, across task formats and prompting strategies?
What are key considerations for designing benchmarks and evaluation frameworks?
Theory-of-Mind (ToM) and Pragmatics
Is there a correlation between LLM capabilities for Theory-of-Mind reasoning and contextual language generation? If so, what are the mechanisms behind these capabilities?
Pragmatics across Cultures and Languages
Are LLM abilities in functional language use similar across cultural contexts and languages?
Application of LLMs for Understanding Human Pragmatic Language Use
How can LLMs be leveraged for better theoretical, experimental and computational understanding of human pragmatic language use?
Interpretability of LLMs' Pragmatic Competence
What are the mechanisms supporting LLMs' pragmatic competence, e.g., through the lens of mechanistic or representational interpretability?

SCHEDULE

9:00 – 9:15 Opening Remarks

9:15 – 10:00 Keynote: Jennifer Hu

10:00 – 10:45 Keynote: Michael Franke

10:45 – 11:15 Coffee Break ☕

11:15 – 12:15 Spotlight Talks 🌟

11:15 - 11:27: The or That? Evaluating language models’ sensitivity to discourse structure in anaphora - Jennifer Hu, Ankana Saha, Kathryn Davidson
11:27 - 11:39: Are Large Language Models Sensitive to the Motives Behind Communication? - Addison J. Wu, Ryan Liu, Kerem Oktar, Theodore Sumers, Thomas L. Griffiths
11:39 - 11:51: You Prefer This One, I Prefer Yours: Using Reference Words is Harder Than Vocabulary Words for Humans and Multimodal Language Models - Dota Tianai Dong, Yifan Luo, Po-Ya Angela Wang, asli ozyurek, Paula Rubio-Fernandez
11:51 - 12:03: Inside you are many wolves: Using cognitive models to interpret value trade-offs in LLMs - Sonia Krishna Murthy, Rosie Zhao, Jennifer Hu, Sham M. Kakade, Markus Wulfmeier, Peng Qian, Tomer Ullman
12:03 - 12:15 : Do Large Language Models Have a Planning Theory of Mind? Evidence from MindGames: a Multi-Step Persuasion Task - Jared Moore, Ned Cooper, Rasmus Overmark, Beba Cibralic, Cameron Robert Jones, Nick Habe

12:15 - 1:15 Lunch

1:15 – 2:00 Keynote: Daniel Fried

2:00 – 2:45 Keynote: Vera Demberg

2:45 – 4:00 Coffee Break ☕ & Poster Session

Sarc7: Evaluating Sarcasm Detection and Generation with Seven Types and Emotion-Informed Techniques - Lang Xiong, Raina Gao, Alyssa Jeong, Yicheng Fu, Kevin Zhu, Sean O'Brien, Vasu Sharma
(RSA)^2: A Rhetorical-Strategy-Aware Rational Speech Act Framework for Figurative Language Understanding - Cesare Spinoso-Di Piano, David Eric Austin, Pablo Piantanida, Jackie CK Cheung
PoliRat: Investigation of Politeness Principles in Large Language Models using Rationales - Ritam Dutt, Carolyn Rose
Investigating the use of pragmatic inferences and the predictive power of language models in sentence processing - Dingyi Pan, Andrew Kehler
On the Same Wavelength? Evaluating Pragmatic Reasoning in Language Models across Broad Concepts - Linlu Qiu, Cedegao E. Zhang, Joshua B. Tenenbaum, Yoon Kim, Roger P. Levy
GPT-2’s comprehension and production of scope ambiguity - Shiva Upadhye, Yanting Li, Jiaxuan Li, Noa Attali, Gregory Scontras
IRONIC: Coherence-Aware Reasoning Chains for Multi-Modal Sarcasm Detection - Aashish Anantha Ramakrishnan, Aadarsh Anantha Ramakrishnan, Dongwon Lee
Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges - Bolei Ma, Yuting Li, Wei Zhou, Ziwei Gong, Yang Janet Liu, Katja Jasinskaja, Annemarie Friedrich, Julia Hirschberg, Frauke Kreuter, Barbara Plank
Pragmatic Generalization in LLMs: Insights from Fine-Tuning and Evaluating on Multilingual Sarcasm - Girma Yohannis Bade, Olga Kolesnikova, José Luis Oropeza
On Language Models' Sensitivity to Suspicious Coincidences - Sriram Padmanabhan, Kanishka Misra, Kyle Mahowald, Eunsol Choi
Social Scaffolds: A Generalization Framework for Social Understanding Tasks in Conversations - Ritam Dutt, Carolyn Rose, Maarten Sap
Emotional RAG LLMs: Reading Comprehension for the Open Internet - Benjamin Reichman, Adar Avsian, Larry Heck
The Pragmatic Mind of Machines: Tracing the Emergence of Pragmatic Competence in Large Language Models - Kefan Yu, Qingcheng Zeng, Weihao Xuan, Wanxin Li, Jingyi Wu, Rob Voigt

4:00 – 5:00 Panel Discussion with Keynote Speakers

5:00 - 5:10 Closing Remarks 👋

SUBMISSION FORMAT

We seek both 4-page extended abstracts and 9-page full papers, excluding references and appendices. All workshop papers are non-archival, and we welcome position papers on topics of interest to the workshop.
All submissions will be in the COLM 2025 LaTeX format and submitted via the OpenReview portal. Accepted papers will be invited for poster/oral presentations and will be publicly available on the workshop website.