Katy Milkman on"The Power and Pitfalls of Megastudies for Advancing Applied Behavioral Science"

Katy Milkman (University of Pennsylvani)

Title: "The Power and Pitfalls of Megastudies for Advancing Applied Behavioral Science"

May, 15th, 16:00 CET (GMT+1)

Registration: tba

Abstract

Increasingly, policymakers are turning to behavioral science for insights about how to improve citizens’ decisions and outcomes. However, these insights can only inform policy insofar as they are comparable—and unfortunately, different intervention ideas are typically tested across different samples on different outcomes over different time intervals. In my talk, I will introduce the “megastudy,” a massive field experiment in which the effects of many different interventions are compared in the same population on the same objectively measured outcome for the same duration.  I will then share results from four megastudies that my team has conducted over the last several years to illustrate the power and pitfalls of this methodology. I will describe a megastudy targeting physical exercise among 61,293 members of an American fitness chain in which 30 scientists worked in small, independent teams to design a total of 54 different four-week digital programs encouraging exercise. I will then describe a megastudy encouraging in-pharmacy flu vaccinations among 689,693 Walmart pharmacy patients in which 44 scientists worked in teams to design a total of 22 different text reminders using a variety of different behavioral science principles.  Next, I will discuss a megastudy encouraging flu vaccination at doctor’s appointments among 47,306 patients of two large U.S. health systems in which 42 scientists worked in teams to design a total of 19 different text reminders using a variety of different behavioral science principles.  Finally, I will discuss a megastudy encouraging COVID-19 bivalent booster vaccination among 3.5 million patients of a national pharmacy chain in which 13 scientists worked together to design a total of 8 different interventions including text reminders and free round-trip rides to the pharmacy.  I will discuss the accuracy of experts’ and laypeople’s forecasts of the performance of different interventions tested in these studies, share best practices for running megastudies, and describe some key limits of this approach to applied behavioral science research.