NLPOR @ COLM 2025

Welcome to the First Workshop on Bridging NLP and Public Opinion Research!

co-located with COLM 2025, October 10, 2025, Montreal, Canada

🛎️ NEWS: Accepted submissions are now available for view on here (direct submission) and here (ARR commitment)!

NLPOR aims to strengthen the emerging connection between NLP and public opinion research (POR);

joining forces to improve both data collection and human-facing technology.

Workshop Aims and Scope

Advances in NLP and Large Language Models (LLMs) are driving changes in how we engage with and interpret human-generated data. The NLP community is increasingly interested in LLM evaluation (e.g., Sun et al., 2023, Röttger et al., 2024), language model pretraining research (Gao et al., 2020, Soldaini et al., 2024), data-centric approaches to data collection (Gebru et al., 2021, Paullada et al., 2021), data ethics (Koch et al., 2021) and biases (Gallegos et al., 2024). Progress in quantifying data quality (Swayamdipta et al., 2020) and mitigating biases (Bender & Friedman, 2018, Srivastava 2020) will require rigorously defined methods and robust measures. Public opinion research (POR), the science of collecting and analyzing high-quality data from and about humans, can help move this research forward. Closer collaboration between the two fields has immense potential for mutual benefit, combining the rigorous data collection methods from public opinion research with the speed and insights of NLP tools.

How POR can improve NLP: As LLMs and generative AI broadly are integrated into human-facing applications, it is increasingly critical that these models are grounded in high-quality, representative data. Public opinion researchers specialize in designing instruments to capture accurate, representative data from human subjects. NLP researchers can draw on this expertise to improve the quality of data used to train, fine-tune, and evaluate LLMs (Durmus et al., 2024), which can increase model performance (Kern et al., 2023). POR's focus on data accuracy, the cognitive response process, and bias mitigation can help improve the performance and fairness of LLMs.
How NLP can improve POR: NLP also offers tools and techniques that can make collection of public opinion data more efficient and more accurate. For example, LLMs can extract information from free text responses (He et al., 2024), serve as interviewers (Xiao et al., 2020, Kim et al., 2019), write survey questions (Jansen et al., 2023), and impute missing data (Callegaro and Yang, 2018). LLMs can also answer survey questions, serving as synthetic responses to test questionnaires (Jordan et al., 2022, Argyle et al., 2023). The careful and responsible integration of NLP into POR workflows can enhance data quality, efficiency, and analysis.

NLPOR is timely and critical as LLMs increasingly influence public discourse, decision-making, and social science research. By fostering collaboration, this workshop will help both fields tackle key challenges in data ethics, bias mitigation, and methodological transparency in the era of LLMs.

Topics of Interest

NLPOR welcomes submissions (not limited) on:

High-Quality Data: How to obtain/what is its impact? Data-centric NLP/AI, statistical and social science theory informed data collection, e.g., practical guidance on sampling,, annotation instrument design, annotator selection and training, annotation errors vs. variations vs. subjectivity, data bias, fairness, data quality estimation.
Trustworthy and Reliable (LLM) Evaluation: How to reliably evaluate NLP/LLMs to increase trust? Protocols for human evaluation of LLMs and evaluation involving human subjects, ethical considerations, hybrid human-LLM evaluation, trust, actionable evaluation protocols and interpretability (e.g. behavioral testing), etc.
Use of LLMs and NLP in POR: Position papers, successes and failures using LLMs to develop or review questions, analyze open-ended responses, generate synthetic responses, impute missing values, etc. How and when to use LLMs and NLP?
Training and alignment of LLMs to represent opinions: Effects of pre-training and fine-tuning techniques and data to ensure models represent diverse opinions and personas.