Using LLMs for Computational Social Science
Diyi Yang
Stanford University
@diyi_yang
Caleb Ziems
Stanford University
@cjziems
Niklas Stoehr
ETH Zurich
@niklas_stoehr
July 17, 13:30-17:00
Houston: Ben Franklin Room (B)
Schedule
13:30-13:40 Introduction (Diyi)
13:40-14:10 LLMs for Measurement (Caleb)
14:10-15:40 Technical and Practical Considerations (Niklas)
14:40-15:00 Coding Demo (Niklas)
15:00-15:30 QA and Coffee Break
15:30-15:50 Hypothesis Generation (Caleb)
15:50-16:30 Simulation + Intervention (Diyi)
16:30-16:50 Coding Demo (Caleb)
16:50-17:00 Conclusion (Diyi)
Abstract
Abstract
Our tutorial will guide participants through the practical aspects and hands-on experiences of using Large Language Models (LLMs) in Computational Social Science (CSS). In recent years, LLMs have emerged as powerful tools capable of executing a variety of language processing tasks in a zero-shot manner, without the need for task-specific training data. This capability presents a significant opportunity for the field of CSS, particularly in classifying complex social phenomena such as persuasiveness and political ideology, as well coding or explaining new social science constructs that are latent in text. This tutorial provides an in- depth overview on how LLMs can be used to enhance CSS research. First, we will provide a set of best practices for prompting LLMs, an essential skill for effectively harnessing their capabilities in a zero-shot context. This step of the talk assumes no prior background. We will explain how to select an appropriate model for the task, and how factors like model size and task complexity can help researchers anticipate model performance. To this end, we introduce an extensive evaluation pipeline, meticulously designed to assess the performance of different language models across diverse CSS benchmarks. By covering these results, we will show how CSS research can be broadened to a wider range of hypotheses than prior tools and data resources could support. Second, we will discuss some of the limitations with prompting as a methodology for certain measurement scales and data types, including ordinal data, and continuous distributions. This part will look more “under the hood” of a language model to outline challenges around decoding numeric tokens, probing model activations as well as intervening on model parameters. By the end of this session, attendees will be equipped with the knowledge and skills to effectively integrate LLMs into their CSS research.
Our tutorial will guide participants through the practical aspects and hands-on experiences of using Large Language Models (LLMs) in Computational Social Science (CSS). In recent years, LLMs have emerged as powerful tools capable of executing a variety of language processing tasks in a zero-shot manner, without the need for task-specific training data. This capability presents a significant opportunity for the field of CSS, particularly in classifying complex social phenomena such as persuasiveness and political ideology, as well coding or explaining new social science constructs that are latent in text. This tutorial provides an in- depth overview on how LLMs can be used to enhance CSS research. First, we will provide a set of best practices for prompting LLMs, an essential skill for effectively harnessing their capabilities in a zero-shot context. This step of the talk assumes no prior background. We will explain how to select an appropriate model for the task, and how factors like model size and task complexity can help researchers anticipate model performance. To this end, we introduce an extensive evaluation pipeline, meticulously designed to assess the performance of different language models across diverse CSS benchmarks. By covering these results, we will show how CSS research can be broadened to a wider range of hypotheses than prior tools and data resources could support. Second, we will discuss some of the limitations with prompting as a methodology for certain measurement scales and data types, including ordinal data, and continuous distributions. This part will look more “under the hood” of a language model to outline challenges around decoding numeric tokens, probing model activations as well as intervening on model parameters. By the end of this session, attendees will be equipped with the knowledge and skills to effectively integrate LLMs into their CSS research.
Links to Resources