May 12th, 2024
LLMs as Research Tools:
Applications and Evaluations in HCI Data Work
a workshop at CHI 2024
Overview
Large language models (LLMs) stand to reshape traditional methods of working with data. While LLMs unlock new and potentially useful ways of interfacing with data, their use in research processes requires methodological and critical evaluation.
In this workshop, we seek to gather a community of HCI researchers interested in navigating the responsible integration of LLMs into data work: data collection, processing, and analysis.
We aim to create an understanding of how LLMs are being used to work with data in HCI research, and document the early challenges and concerns that have arisen. Together, we will outline a research agenda on using LLMs as research tools to work with data by defining the open empirical and ethical evaluation questions and thus contribute to setting norms in the community.
We believe CHI to be the ideal place to address these questions due to the methodologically diverse researcher attendees, the prevalence of HCI research on human interaction with new computing and data paradigms, and the community's sense of ethics and care. Insights from this forum can contribute to other research communities grappling with related questions.
Key Information
Submission Deadline:
Closed to submissions
(Individuals still welcome to send expressions of interest)
February 26th, 2024, AOE
Notification Date:
March 22nd, 2024
Camera-ready deadline:
May 1st, 2024
Remote-first participation sessions:
May 8th at 8am PT/11am ET
Hybrid Workshop:
Sunday, May 12, 2024
👉 Submission link 👈
Workshop Goals
Goal #1:
Bring the community together to discuss, reflect, and share ongoing applications and challenges of using LLMs to work with data in HCI research
Goal #2:
Discuss options for establishing methodological validity when using LLMs to work with data in HCI research
Goal #3:
Discuss the primary critical and ethical questions regarding the use of LLMs to work with data in HCI research
Call for Participation
Broadly accessible large language models (LLMs) stand to fundamentally reshape the HCI community’s suite of methods for working with data. To date, LLM tools have already been used to facilitate qualitative coding, perform thematic analysis, and even mediate interviews or simulate user data.
However, we lack a broader understanding of:
How LLM-based methods are being used to work with data in HCI
What empirical evaluation strategies are acceptable to the community for establishing validity of data work conducted with LLMs
How to critically and ethically use LLM methods in HCI research.
The goal of this workshop is to gather a community of researchers interested in these topics to map current approaches as a community, documenting the challenges encountered, and norm-set in this rapidly evolving field.
For this hybrid CHI 2024 workshop, we invite junior and senior academics, researchers, and practitioners to submit extended abstracts or short papers. Interested participants should submit a 2-4 page (not including references) proposal via this link using the CHI Extended Abstracts format.
We invite submissions including empirical works-in-progress, research or research proposals, and provocations, critical approaches, or position papers. Broadly, paper topics should relate to the use of LLMs to work with data in HCI, epistemic validity and methodological evaluations, and/or critical and ethical perspectives on the use of LLM methods in HCI research. One participant from each submission must register for the workshop and at least one day of the conference. With author consent, submissions will be published on the workshop website.
We welcome perspectives on applications, as well as methodological and critical evaluation from researchers of different methodological backgrounds, including NLP, qualitative research, user research, and beyond.
Topics
Topics under consideration may include, but are not limited to:
Approaches to using LLM-based methods to work with data in HCI
LLM involvement in the generation of qualitative codebooks or in applying (classifying) tags for thematic analysis
LLM involvement in survey-generation or deployment
Simulation of response data or user interactions
LLMs to synthesize, summarize, or sensemaking with interview, text, or other unstructured data
Epistemic validity and community norms
Proposed empirical evaluation strategies for establishing validity of data work conducted with LLMs
Approaches using or reflecting on standards from NLP (e.g. benchmarks) or qualitative social research (e.g. interrater reliability) to validate LLM labeling and classification
Validity and norms of LLMs compared with prior methodological changes, such as crowdwork for data labeling or automated transcription of interviews
Critical and ethical perspectives of the use of LLM methods in HCI research
Most pressing risks or harms to surface and address
How to weigh risks and harms relative to potential usefulness
Approaches to make harms and risks visible or correct them
Organizers
Cornell
Rutgers
Singapore University of Technology and Design
Stanford
Singapore University of Technology and Design
Questions? Please contact msa258@cornell.edu