Prof Robb Willer, Stanford
YouTube Stream: https://www.youtube.com/watch?v=RaL1PyctHr8
Join group to receive calendar invite: https://groups.google.com/a/modelingtalks.org/g/talks
Abstract:
Randomized experiments are the best tool social scientists have for learning what changes people’s attitudes and behavior, but they are slow and expensive, so most ideas never get tested. In this talk I ask whether large language models can forecast the results of experiments before they are run. We assembled an archive of 70 preregistered, nationally representative U.S. survey experiments covering 469 treatment effects and about 119,000 participants. We then prompted an LLM to respond to each experiment’s materials as a demographically representative sample of simulated Americans, and estimated treatment effects by comparing simulated responses across conditions. Predictions from GPT-4 correlated strongly with the real effects (r = 0.85), about as well as pooled human forecasts. Accuracy held up for studies that had not been published by the model’s training cutoff, and for open-weight models. The method has clear limits. Predicted effects are roughly two to three times too large, and accuracy is lower in a second archive of 15 megastudies, particularly for field experiments, though still comparable to expert forecasters. I will discuss how researchers say they would use these forecasts, including power analysis and screening interventions before a costly study, along with risks such as bias and misuse, which we tested directly. Time permitting, I will also describe related work building generative agents of 1,052 real individuals from two-hour interviews. These agents reproduce people’s survey answers 85% as accurately as the people reproduce their own answers two weeks later.
Bio: Robb Willer is Professor of Sociology and, by courtesy, of Psychology and Organizational Behavior at Stanford University, a 2026 Guggenheim Fellow, and the director of the AI for Public Benefit Lab and the Politics and Social Change Lab. Willer studies the use of artificial intelligence for social science and public benefit, including how large language models can forecast the results of experiments and policy interventions, simulate the attitudes and behavior of individual people, give voters accurate, nonpartisan election information, and support emotional well-being. In a 2026 Nature paper, he and his colleagues showed large language models can accurately predict the results of social science experiments, including studies the models could not have encountered in training. His research on AI persuasion, credited in Science as the first demonstration that generative AI could write policy messages capable of persuading humans, has informed how leading AI companies evaluate their models. He also studies polarization, persuasion, and democracy, and co-organized the Strengthening Democracy Challenge, a megastudy published in Science that tested 25 interventions on more than 32,000 Americans. He has published more than 100 articles, including in Science, Nature, the Proceedings of the National Academy of Sciences, and Nature Human Behaviour.
Bio:
Robb Willer is Professor of Sociology and, by courtesy, of Psychology and Organizational Behavior at Stanford University, a 2026 Guggenheim Fellow, and the director of the AI for Public Benefit Lab and the Politics and Social Change Lab. Willer studies the use of artificial intelligence for social science and public benefit, including how large language models can forecast the results of experiments and policy interventions, simulate the attitudes and behavior of individual people, give voters accurate, nonpartisan election information, and support emotional well-being. In a 2026 Nature paper, he and his colleagues showed large language models can accurately predict the results of social science experiments, including studies the models could not have encountered in training. His research on AI persuasion, credited in Science as the first demonstration that generative AI could write policy messages capable of persuading humans, has informed how leading AI companies evaluate their models. He also studies polarization, persuasion, and democracy, and co-organized the Strengthening Democracy Challenge, a megastudy published in Science that tested 25 interventions on more than 32,000 Americans. He has published more than 100 articles, including in Science, Nature, the Proceedings of the National Academy of Sciences, and Nature Human Behaviour.